Systolic array with efficient input reduction and extended array performance

ABSTRACT

Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reduced input can include a reduced input data element and/or a reduced weight. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide the reduced input to the array. In order to reduce the bit-length, the reducer may reduce the number of trailing bits of the input. Further, the systolic array can receive a reduced and rounded input. The systolic array can propagate the reduced input through the processing elements in the systolic array. Each processing element may include a multiplier and/or an adder to perform arithmetical operations based on the reduced input.

BACKGROUND

Artificial neural networks are computing systems with an architecturebased on biological neural networks. A neural network may be implementedby circuitries and data paths, such as a systolic array. Systolic arrayscan accelerate the performance of the training and inference phases ofartificial neural networks. During the training phase, input data can beprovided to train a model. During the inference phase, new inputs can beprocessed according to the model to obtain a predicted result. Userapplications often use the model in the inference phase, so theinference phase can often have time sensitivities, and latency duringthe inference phase can negatively impact the user experience.

As more applications use artificial neural networks, the applicationsalso use a wide range of numbers that may include numbers with increasedbit-lengths (e.g., 32-bit floating-point numbers) that may requiregreater computational power or modifications to the neural networks.While computational support for numbers with the increased bit-lengthscan provide increased accuracy for mathematical operations, providingsupport for the increased bit-lengths of these numbers can increase thecomplexity, size and cost of the processing elements in the systolicarray. These increases can also affect the system processing speed andthe system power consumption. Power consumption and the size of thesystolic array can become highly important when a systolic array isrequired to support a wide range of numbers.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features will now be described with reference to the followingdrawings. Throughout the drawings, reference numbers may be re-used toindicate correspondence between referenced elements. The drawings areprovided to illustrate examples described herein and are not intended tolimit the scope of the disclosure.

FIG. 1A illustrates an example 4×4 systolic array and an example columnof reducers.

FIG. 1B illustrates an example 1×8 column of a systolic array.

FIG. 2A illustrates a processing element for neural network computationswith the inputs entering through separate reducers, according to certainexamples of the disclosed technologies.

FIG. 2B illustrates a processing element for neural network computationswith the inputs entering through the same reducer, according to certainexamples of the disclosed technologies.

FIG. 3 illustrates an apparatus including zero detector circuits forreduced input data elements and reduced weights entering a systolicarray for neural network computations, according to certain examples ofthe disclosed technologies.

FIG. 4A illustrates a reducer showing the selection of an input to bereduced and rounded, according to some examples of the disclosedtechnologies.

FIG. 4B illustrates a reducer showing the selection of a rounded inputto be reduced, according to some examples of the disclosed technologies.

FIG. 4C illustrates a reducer showing the generation of multiple reducedinputs from a selected input, according to some examples of thedisclosed technologies.

FIG. 5 illustrates a multiply accumulate datapath for neural networkcomputations, according to certain examples of the disclosedtechnologies.

FIG. 6 shows an apparatus for neural network computations according tosome examples of the disclosed technologies.

FIG. 7 shows a method executed by a reducer and a processing element forneural network computations, according to some examples of the disclosedtechnologies.

FIG. 8 shows a method executed by a reducer and a processing element forneural network computations, according to some examples of the disclosedtechnologies.

FIGS. 9A-9H show an example systolic array processing data over asequence of systolic intervals.

FIG. 10 illustrates an example of a computing device, according tocertain aspects of the disclosure.

DETAILED DESCRIPTION

Generally described, the present disclosure relates to a systolic arraythat supports converting inputs of a higher bit-length than elements ofthe array natively support into one or more reduced inputs. Further, theinput can be converted into a reduced input for single-pass reducedprecision computations on inputs of a higher bit-length than elements ofthe array natively support. For example, the elements of the array maynatively support single-pass computations on inputs of a particularbit-length of the systolic array and the systolic array may receiveinput from a reducer that reduces a bit-length of the input to match thebit-length natively supported by the elements during single-passcomputations. The input may also be converted into multiple reducedinputs for multiple-pass full precision computations on inputs of ahigher-bit length than elements of the array natively support. Asdescribed herein, the use of such a reducer to provide the reduced inputto the systolic array can enable inputs to be given to a systolic arrayin an arbitrary bit-length, and the inputs may be programmaticallyadjusted to a particular bit-length (e.g., a highest bit-lengthsupported during single-pass computations) such that a user need not beaware of the particular bit-length of the inputs to the processingelements of the systolic array. While a traditional systolic array maysupport different bit-lengths, the native support of single-passcomputations for a higher bit-length can increase the size and powerconsumption of a systolic array. Further, this may affect processing ofshorter bit-lengths. Therefore, traditional systolic arrays must balancethe ability to do single pass computations for longer bit-lengths andthe efficiency in processing shorter bit-lengths. This may result insystolic arrays that do not support longer bit-lengths due to a loss inefficiency in processing the shorter bit-lengths. Disclosed herein is asystolic array to support arbitrarily long bit-lengths at a reducedprecision with a minimal loss in efficiency in comparison to processingthe shorter bit-lengths. The systolic array may support inputs ofarbitrary bit-lengths through a reducer that can drop excess bits fromthe significand of the input with an arbitrary bit-length and round theremaining bits. The dropping of the excess bits can enable the reducerto reduce the bit-length of the input to the maximum bit-lengthsupported for single-pass computations by the systolic array, at thecost of reduced precision from the arbitrary bit-length. Further, theuse of such a reducer can enable the systolic array that receives inputsof arbitrary bit-lengths to provide the same performance as achieved bya systolic array that receives inputs of fixed bit-lengths. Allowing auser to provide inputs with arbitrary (or non-fixed) bit-lengths mayallow for lower-cost or lower-power elements to be used in a systolicarray receiving inputs with a greater bit-length, while maintaining theoverall performance of the systolic array due to the reduction in thebit-length of the input by the reducer. Further, by reducing thebit-length of the input (e.g., a 32-bit floating-point number), thereducer can provide a reduced precision version (e.g., a 22-bitfloating-point reduced precision number) of the input. Therefore, thereducer can generate a reduced input from an input by reducing thebit-length of the input.

The reducer can generate multiple reduced inputs from the input. Thesystolic array may utilize the multiple reduced inputs in amultiple-pass multiply-accumulate operation in order to retain theaccuracy of the input. For example, each combination of reduced inputs(e.g., where the reducer generates two reduced inputs for the input dataelement and the weight, input data element 1 and weight 1, input dataelement 2 and weight 1, input data element 1 and weight 2, and inputdata element 2 and weight 2) may be passed through the multiple-passmultiply-accumulate operation. By generating multiple reducing inputswith reduced bit-lengths from the input, the reducer can reduce thebit-length of the input to the maximum bit-length supported forsingle-pass computations by the systolic array, at the cost of reducedperformance from the arbitrary bit-length. Further, the use of such areducer can enable the systolic array that receives multiple reducedinputs (with the bit-length reduced from an original bit-length) toprovide the same frequency, power advantage, and/or size advantage asachieved by a systolic array that receives inputs of fixed (e.g.,standard) bit-lengths at a cost of lower performance as compared to asystolic array that operates on inputs of the original bit-length.Allowing a user to provide inputs with arbitrary bit-lengths may allowfor lower-cost or lower-power elements (e.g., power elements that areconfigured to operate on standard bit-lengths) to be used in a systolicarray receiving inputs with an arbitrary bit-length, while offering anincreased precision as compared to systolic arrays receiving inputs withstandard bit-lengths.

As described herein, a systolic array includes an array of processingelements (PEs), often arranged into two dimensions (e.g., columns androws). The PEs of the array can be interconnected to enable data to passthrough the PEs, which may conduct one or more mathematical operationson the data. For example, each PE may conduct a “multiply accumulate”operation, whereby inputs are fed horizontally into PEs of each row ofthe array, with each PE multiplying its respective input by a storedweight value and passing the product result to a PE in a subsequent row.

One illustrative use of a systolic array is in conducting an inferencephase of a machine learning application. Machine learning generallyrequires at least two phases: a “learning phase,” where a model istrained against training data, and an “inference phase,” in which thetrained model is applied to production data to predict a result.Inference phase applications are often latency sensitive, in that theyoperate in production environments. Moreover, inference phaseapplications—and particularly neural network applications—often requiredense algebraic calculations, such as matrix multiplications. Systolicarrays may be used to accelerate inference-phase workloads in machinelearning applications.

As noted above, the PEs of a systolic array may be divided into rows andcolumns. Each PE in the input layer may receive an element of an inputdata set, and scale the element with a weight (e.g., a filter) toindicate the element's degree of influence on the output. Each PE in theintermediate layers may receive at least one of the element and theweight (or filter) from another PE in the systolic array. Each PE in theintermediate layers may combine the elements received from acorresponding PE of the systolic array to compute a set of intermediateoutputs. For example, each PE in the intermediate layers may compute asum of element-weight products, and then produce the sum for applicationof an activation function to the sum (e.g., by a system separate fromthe PEs of the systolic array).

Generally, an input data set (e.g., an input feature map) may be fed,one input data element at a time, into its respective row of thesystolic array, and passed from one PE to another PE in a given rowstarting, for example, from a leftmost PE. Each row receives a specificinput data element and weight which are fed into a first PE, in a row,and subsequently passed to an adjacent PE located to the right of thefirst PE in the same row. Further, an input partial sum may be fed, oneinput partial sum at a time, into its respective column of the systolicarray, and passed from one PE to another PE in a given column startingfrom a topmost PE. Generally, an input partial sum may be fed from afirst PE, in one column, to an adjacent PE located directly beneath thefirst PE in the same column. Further, each column corresponds to aspecific input partial sum which is passed through each PE of a givencolumn. This can be done to allow each PE of a given column to perform amathematical operation on the input partial sum to produce an outputpartial sum. As the input data element passes through a PE, the inputdata element can be multiplied with the weight value, and accumulatedwith the input partial sum. The first PE, in one column, is provided aninput partial sum and generates an output partial sum based on themathematical operations performed by that PE. The output partial sum isthen provided to an adjacent PE in the same column as an input partialsum. The adjacent PE may then perform further mathematical operationsbefore generating an output partial sum and passing the output partialsum to a further adjacent PE. In some embodiments, input data may be fedinto a systolic array in a cascading fashion, with a PE in a firstcolumn and row (a position that may be designated as [0, 0], indicatingrow and column 0) receiving an input data element and an input partialsum in a first clock cycle. Thereafter, data can generally flow tosubsequent rows and columns at a given rate (e.g., advancing one PE percycle). For example, the output partial sum of the PE at [0, 0] can befed to the PE at [1, 0], along with an input data element for row 1,such that the PE at [1, 0] performs a mathematical operations on thatinput data element and partial sum during a second clock cycle.Similarly, the input data element of PE [0, 0] can be passed to a PE ofa subsequent column (e.g., at position [0, 1]), which can also be fed aninput partial sum, such that the PE at [0, 1] conducts a mathematicaloperation on that input partial sum and input data element during thesecond clock cycle. Assuming a convention in which rows advance downwardand columns advance to the right, data therefore can generally flow downand to the right during operation of the array. To assist in thesecalculations, PEs within the array may be provided with weights prior tothe first clock cycle, or may receive weights in the first clock cycleor during calculations.

As machine learning applications and neural network applicationsproliferate, the demand for increased processing capabilities (e.g., thecapability to handle larger numbers and/or more precise numbers) whileachieving higher precision and maintaining performance has alsoincreased. For example, the demand to support numbers with increasedprecision (e.g., the decimal places for a number and/or the significandfor a number) has increased. Providing support for numbers with greaterbit-lengths (e.g., 32-bit floating-point numbers) results in significantincreases in integrated circuit die cost, power consumption, and circuitcomplexity in comparison to supporting only numbers with fixed (e.g.,particular) bit-lengths (e.g., 16-bit floating-point numbers) as thetraditional PE may not be capable of receiving numbers with bit-lengthsexceeding a particular length. In a systolic array of hundreds orthousands of PEs, the added support for numbers with greater bit-lengthscan cause an exponential increase in the integrated circuit die cost,power consumption, and circuit complexity. In some configurations, a PEsupports performing mathematical operations on numbers with increasedbit-lengths (e.g., 32-bits) with specialized circuitry configured forthe larger bit-lengths. For example, a 32-bit floating-point systolicarray may be specialized to perform mathematical operations on 32-bitfloating-point (FP32) numbers. Such modifications may be particularlyundesirable, may offer reduced performance, and may be costly and/ortime consuming to implement. In other configurations, a PE does notsupport mathematical operations on numbers with bit-lengths exceeding agiven size. For example, a 16-bit floating-point systolic array may notbe capable of performing mathematical operations on numbers other than16-bit floating-point (FP16) numbers. Such a lack of capabilities may beparticularly undesirable and may offer reduced precision and/or reducedprocessing capabilities.

The present disclosure provides a systolic array with significantadvantages over prior implementations. The present disclosure enables asystolic array to support arbitrary bit-lengths and maintain performancefor shorter bit-lengths relative to an array that natively supportssingle-pass computations on longer-bit lengths, without significantlyincreasing power consumption of the array. Moreover, the presentdisclosure can enable the use of numbers with arbitrary bit-lengths(e.g., 32-bit floating-point numbers) as input to the systolic array(e.g., as input to a reducer of the array). Further, the reducer of thesystolic array can programmatically adjust the inputs to a particularbit-length (e.g., a highest bit-length supported during single-passcomputations) such that a user need not be aware of the particularbit-length of the inputs that the processing elements of the systolicarray receive. These advantages are provided by the embodimentsdiscussed herein, and specifically by creation of a systolic arrayutilizing one or more reducers that reduce one or more inputs to beprovided to the systolic array. Further, the one or more reducers cangenerate multiple reduced inputs for a particular input in order toretain the accuracy of the original input.

The systolic array may support particular bit-lengths or data types. Forexample, the systolic array may support standard bit-lengths and/or datatypes (e.g., FP16 numbers). A consumer or user may be notified that thesystolic array supports the particular bit-lengths or data types.Further, the reducer may receive inputs with arbitrary bit-lengths thatdo not correspond to the supported bit-lengths and/or data types (e.g.,FP32 numbers). The reducer may convert the input with a non-supportedbit-length into a reduced format (e.g., a reduced bit-length) andprovide the input with the reduced format (e.g., 22-bit floating-pointnumbers) to the systolic array. The reduced format may be a non-standardformat, a non-standard bit-length, and/or a non-standard data type. Theconsumer may not be notified that the systolic array supports inputswith the reduced format. Further, the input with the reduced format mayhave a higher accuracy or precision than inputs with the standardbit-lengths and/or data types and a higher performance than inputs withthe arbitrary bit-lengths and/or data types as the arbitrary bit-lengthsand/or data types may require specialized software and/or hardware touse these numbers. Further, the internal structure of the systolic arraymay be a superset of the components of each supported data type. Forexample, the internal structure of the systolic array may support astandard significand bit-length from A to B and a standard exponentbit-length from X to Y. Therefore, the maximum internally supportedbit-length of the array may be 1+B+Y, where B and Y may be any number.Further, 1+B+Y may not correspond to a standard format (e.g., 1+B+Y maycorrespond to a 22-bit format) but the reducer may be able to downsizeto this format for input to the array. Therefore, while a set of datatypes and/or bit-lengths may be exposed to the customer as supported bythe systolic array, the reduced format (e.g., an intermediate bit-lengthbetween the arbitrary bit-lengths and the standard bit-lengths) may notbe exposed to the customer and may correspond to a maximum format (e.g.,bit-length) supported by the systolic array. This can enable anincreased accuracy relative to inputs with standard bit-lengths and anincreased performance relative to inputs with arbitrary bit-lengths.

As disclosed herein, each reducer (e.g., bit reducer, zeroer, etc.)assigned to a particular row of the systolic array may reduce one ormore inputs (e.g., change one or more bits to zero) provided to thereducer and output one or more reduced inputs based at least in part onthe one or more inputs. The provided inputs to the reducer may benumbers represented by a significand and an exponent. For example, theprovided inputs may be in floating-point format. The one or more reducedinputs may be represented in a modified format with a reducedsignificand and an expanded exponent. The reduced input may have a signbit, exponent bits, and significand bits. The most significant bit ofthe significand bits may be implied or hidden. Each reducer may includeone or more of the following: a rounder, an exponent expander, atrailing bit reducer, and a multiplexer. The reducer can adjust theinputs provided to the reducer by maintaining the exponent of theoriginal input and reducing the significand of the original input. Thereducer may utilize the rounder to round the reduced input generated bythe reducer based on the unreduced number. In some embodiments, theinput may be pre-rounded to a given precision (e.g., the number of bitssupported for single-pass computations) and the reducer can drop theresulting, trailing zeros to generate the reduced input. The rounder mayuse various rounding techniques to round the input (e.g., any standardrounding technique). Further, the reducer may utilize the exponentexpander to expand a quantity of bits of an exponent portion of thenumber and the trailing bit reducer to reduce the quantity of bits of asignificand portion of the number. Each reducer may contain anycombination of these components. Each reducer may utilize the componentscontained in the reducer to produce a reduced input and provide thereduced input to the systolic array or the processing elements of thesystolic array. By producing the reduced input, the reducer is enabledto reduce or adjust arbitrary bit-lengths (e.g., arbitrarily longbit-lengths) to bit-lengths supported during single-pass computations bythe processing elements of the array, with a loss of precision from theoriginal input of the arbitrary bit-length.

The reducer, by dropping bits and providing a single-pass computationthrough the systolic array, may lead to reduced precision (e.g.,corresponding to the data of the dropped bits). For example, the finaloutput may be a reduced output equal to the reduced weight times thereduced input data element. This precision may be recaptured byimplementing additional passes through the array. For example, thereducer may convert a weight into a high reduced weight and a lowreduced weight and an input data element into a high reduced input dataelement and a low reduced input data element. Further, the final outputmay include greater precision and may equal the low reduced weightmultiplied by the low reduced input data element plus the low reducedweight multiplied by the high reduced input data element plus the highreduced weight multiplied by the low reduced input data element plus thehigh reduced weight multiplied by the high reduced input data element.While the multiple-pass computations may require a reduction in speed(e.g., based on the multiple passes through the array for a single totaloutput), the multiple-pass computations may offer significant increasesin precision over the single-pass computation for reduced precision.Therefore, the systolic array may be able support higher bit-lengthswith hardware that natively supports a maximum bit-length that is lowerthan the higher bit-lengths by receiving inputs from reducers. Eachreducer assigned to a particular row of the systolic array can receive aparticular input data element and/or weight and generate multiplereduced inputs from the received input for multiple passes through(e.g., in) the systolic array for the original input. For example, thereducer can receive an input data element and generate multiple reducedinput data elements based on the input data element in order to retainmore precision of the original input data element as compared toreducing the input to a standard bit-length. The multiple reduced inputsmay sum to generate the input. It will be understood that each input maybe converted into any number of reduced inputs. The reducer may generatethe reduced inputs as a first reduced input (e.g., a high reduced input)and a second reduced input (e.g., a low reduced input). The firstreduced input may be based on the higher magnitude significand bits ofthe input and the second reduced input may be based on the lowermagnitude significand bits. For example, the first reduced input may bebased on the leftmost bits of the significand (e.g., the bits with thehighest magnitude) and the second reduced input may be based on therightmost bits of the significand (e.g., the bits with the lowestmagnitude). Further, the significand of the input may be divided betweenthe first reduced input and the second reduced input. For example, for a23-bit significand, the first reduced input may be based on the first11-bits of the significand as read from left to right (e.g., bits 22 to12) and the second reduced input may be based on the next 12-bits of thesignificand as read from left to right (e.g., bits 11 to 0).

The reducer may generate the first reduced input by zeroing a number oflow bits of the original input. Further, the reducer may generate thesecond reduced input by zeroing a number of high bits of the originalinput. In some embodiments, the reducer may determine that the input isa normal (e.g., not a denormal or subnormal) number by removing animplicit leading bit and renormalizing the reduced significand (e.g.,the significand after zeroing the number of leading bits). The reducermay renormalize the input by shifting the significand a number of bitsbased on a number of leading zeroes. For example, a leading one of thereduced significand may be shifted into the implied bit position. Thereducer may further adjust the exponent based on the number of bitsshifted by the reducer. As adjusting the exponent may cause the exponentto be outside of the range of the current exponent, the reducer mayexpand the exponent (e.g., from 8-bits to 9-bits) such that the adjustedexponent can be represented in the expanded exponent. For example, therange of an 8-bit exponent may enable an exponent value between −126 and+127 and by expanding the exponent to a 9-bit exponent the reducer mayenable an exponent value between −254 to +255. As renormalizing a 32-bitinput may require an exponent as low as −149 (−126−23) to allow shiftingacross the full 23 bits of significand (e.g., where the exponent is“00000000” and the significand is “00000000000000000000001”), thereducer may therefore expand the 8-bit exponent of the input to generatethe second reduced input. The reducer may expand the exponent of thefirst reduced input and the second reduced input. In some embodiments,the reducer may only expand the exponent of the second reduced input.

Each of the first reduced input and the second reduced input may berepresented with a reduced (e.g., compressed) format (e.g., a 21-bitlength). One or more reducers may produce reduced inputs for the inputdata element and the weight. The one or more reducers may furtherprovide each combination of the reduced inputs to the systolic array forthe multiply-accumulate operations. The systolic array may implementmultiple-pass multiply-accumulate operations for the combinations of thereduced inputs to generate a total output. For example, themultiply-accumulate operations may be performed on a first reducedweight and a first reduced input data element, a first reduced weightand a second reduced input data element, a second reduced weight and afirst reduced input element, and a second reduced weight and a secondreduced input data element. For example, the final output may be equalto the first reduced weight multiplied by the first reduced input dataelement plus the first reduced weight multiplied by the second reducedinput data element plus the second reduced weight multiplied by thefirst reduced input data element plus the second reduced weightmultiplied by the first reduced input data element. An adder can sum theoutput of each multiply-accumulate operation (e.g., each partialmultiply-accumulate operation) to generate the total output. Bygenerating the multiple reduced inputs (e.g., inputs with reducedbit-lengths) from an input (e.g., an input with an arbitrarybit-length), the systolic array may be able to performmultiply-accumulate operations on the input (multiple reduced inputversions of the input) without being required to support the arbitrarybit-length of the input. The systolic array may have certain frequencyconstraints, size constraints, etc. in order to maintain performancegoals. In light of these constraints, traditional systolic arrays may beunable to support arbitrary bit-lengths. By generating multiple reducedinputs for a particular input, the systolic array may satisfy theseconstraints while generating outputs based on inputs with arbitrarybit-lengths. It will be understood that any number of reduced inputs maybe generated from an original input. For example, a 64-bitfloating-point number may be converted into 5 21-bit reducedfloating-point numbers. Each of the reduced inputs may correspond to aportion of a significand portion of the original input. For example, afirst reduced input may correspond to a first portion of the significandportion of the original input, a second reduced input may correspond toa second portion of the significand portion, a third reduced input maycorrespond to a third portion of the significand portion, etc. Theparticular portion of the significand portion of the original input fora particular reduced input may be identified by zeroing other portionsof the significand portion.

In some embodiments, the reducer may contain or receive a signal from amultiplexer that selects among two or more inputs based on a controlsignal, such as an opcode or a data type indicator. For example, themultiplexer may identify a particular input for reduction (e.g., aweight or an input data element).

In some embodiments, a systolic array can have separate reducers thatreceive one of either the input data element or the weight and providethe corresponding reduced version of that input to the systolic array.Each processing element in the initial column of processing elements ofthe systolic array may receive multiple reduced inputs from one or morereducers. For example, a first processing element of the initial columnmay receive a reduced input data element from a first reducer and areduced weight from a second reducer and a second processing element ofthe initial column may receive a reduced input data element from a thirdreducer and a reduced weight from a fourth reducer.

Each reducer may reduce the bit-length of numbers of 16-bits, 32-bits,or any number of bits. For example, a reducer may reduce the bit-lengthof a 32-bit floating-point number to a 22-bit floating-point number. Inone embodiment, the 32-bit floating-point number has a 1-bit sign, an8-bit exponent, and a 23-bit significand. From such a 32-bitfloating-point number, the reducer may generate a reduced 20-bitfloating-point number with a 1-bit sign, an 8-bit exponent, and an11-bit significand. In some embodiments, the reducer may increase thebit-length of the exponent of the input in order to adjust the format ofthe reduced input to a format supported by the processing element. Forexample, the reducer can increase the exponent from 8 bits to 10 bits.In some embodiments, in order to reduce the bit-length of a particularnumber, the reducer can reduce a quantity of trailing bits of thesignificand of the number (e.g., the reducer can zero the low bits ofthe significand of the number). For example, the number may be a binarystring “10101010101111111111111” and the reducer may zero the twelvetrailing bits of the number to generate a reduced binary string“10101010101000000000000” and/or “10101010101.”

Each reducer may further round the resulting reduced input to thesystolic array. The reducer can round the reduced input to a particularprecision or number of bits supported by the processing elements of asystolic array. For example, a reducer can round a number to generate arounded number. By rounding the input to the systolic array, thesystolic array can obtain a higher accuracy result for the calculationsof the systolic array. In some embodiments, the reducer can round thereduced input. In other embodiments, the reducer can receive a roundedinput (e.g., an input rounded by a separate system) and reduce therounded input. The rounding may include one or more of stochasticrounding, rounding to nearest even, rounding to zero, rounding down, orrounding up. Further, a user, system, etc. may specify the roundingmethod for rounding the input (e.g., via a selection from a userinterface).

The systolic array may have PEs that each include a 22-bit multiplierand a 34-bit adder. The 22-bit multiplier may operate on 22-bit reduced,floating-point numbers reduced by the reducer from 32-bit floating-pointnumbers to generate a multiplier product with a sign bit, ten exponentbits, and 23 significand bits. The multiplier product may include 24significand bits where the most significant bit is implied or hidden.The 34-bit adder may operate on 34-bit numbers (e.g., the 34-bitmultiplier product). Further, the adder may operate on 35-bit numberswhere one bit is implied or hidden. In some embodiments, the systolicarray may be include an n-bit multiplier and an m-bit adder wherein nmay be any number and the n-bit multiplier and m-bit adder may beoperate on x-bit reduced floating-point numbers. The variables n, m, x,and y may be any number where n is greater than x.

In the following description, various examples will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the examples.However, it will also be apparent to one skilled in the art that theexamples may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe examples being described.

FIG. 1A illustrates an example 4×4 systolic array 100A. The systolicarray 100A illustratively includes four columns of PEs and four rows ofPEs with four PEs in each row, and four PEs in each column. It will beunderstood that the systolic array 100A is simplified for the purpose ofdescription, and that a systolic array 100A in accordance with thepresent disclosure may include any number of PEs in each row and column.Further, the number of PEs in each row may be different than the numberof PEs in each column. It will be further understood that such asystolic array 100A may be logically organized in any number of rows andany number of columns. Further, the number of rows may be different thanthe number of columns. The systolic array 100A may be part of a neuralnetwork processor in a computer system. For example, the computer systemmay provide multi-tenant compute services for data processingapplications such as an image recognition service, text-based dataprocessing (e.g., processing of search queries), audio or video dataprocessing, etc.

Each PE may include a respective row input bus 102, a respective columninput bus 104, a respective column output bus 106, and a respective rowoutput bus 108. A PE may receive inputs from a left PE of the same row(or from external circuitries) via the row input bus 102. The PE mayalso receive inputs from a PE of the same column above (or from externalcircuitries) via the column input bus 104. The PE may perform arithmeticcomputations based on the inputs, and transmit the result of thearithmetic computations to a PE of the same column below (or to theexternal circuitries) via the column output bus 106. The PE may alsoforward the inputs received via the row input bus 102 to a right PE ofthe same row via the row output bus 108.

The systolic array 100A may perform arithmetic computations, includingmultiplication and addition operations, for the processing elements of aneural network. For example, each PE may include arithmetic units suchas a multiplier and an adder. In some embodiments, the multiplier andthe adder may be a fused multiplier adder. In the example of FIG. 1A,each row of the PEs may handle one set of input data, and each column ofthe PEs may generate one set of output data based on the sets of inputdata received by each PE in a given column.

A column 112 of the PEs (the leftmost column) may receive four sets ofinput data, with each set of input data being handled by one row of thePEs. A column 116 of reducers may provide four sets of reduced inputdata to the column 112 of the PEs, with each set of input data beingprovided by one reducer which can increase the overall performance ofthe array as compared to traditional arrays. It will be understood thatthe column 116 of reducers may provide any number of sets of reducedinput to the column 112 of the PEs. For example, the number of reducersand/or the number of sets of reduced input may be based on a quantity ofPEs in a given column. In the example of FIG. 1A, the column 112 of thePEs includes four PEs (PE 112 a, PE 112 b, PE 112 c, PE 112 d) and thecolumn 116 of reducers include four corresponding reducers (reducer 116a, reducer 116 b, reducer 116 c, reducer 116 d). It will be understoodthat the column 116 of reducers may include any number of reducers. Eachreducer in the column 116 of reducers may provide a set of reduced inputdata for a particular PE of the column 112 of PEs, wherein each set ofreduced input data includes two or more reduced inputs. For example, thereducer 116 a may provide a reduced input data element and a reducedweight to the PE 112 a. Each reducer in the column 116 of reducers mayconvert the inputs into reduced inputs. For example, the reducer 116 amay convert a 32-bit input data element into a reduced 22-bit input dataelement.

Each reducer in the column 116 of reducers may further select a reducedinput to provide to each PE in the column 112 of the PEs. For example,each reducer in the column 116 of reducers may contain a multiplexer toselect a reduced weight or a reduced input data element to provide tothe PE. In some embodiments, each reducer 116 a-116 d may be implementedas multiple reducers (e.g., a first reducer and a second reducer).Further, the first reducer and the second reducer may provide one ormore inputs to the column 112 of the PEs. For example, a first reducerof the reducer 116 a may provide a reduced input data element to the PE112 a and a second reducer of a reducer 116 a may provide a reducedweight to the PE 112 a. In some embodiments, a PE may receive a reducedinput (e.g., a reduced input data element) and a non-reduced input(e.g., a non-reduced weight) for arithmetic operations.

Each PE in the column 112 may obtain, from the corresponding input dataset received via the row input bus 102, the reduced input data elementand the reduced weight. Each PE in the column 112 may multiply thereduced input data element with the reduced weight to generate a scaledinput. The scaled inputs generated by the PEs within any column(including the column 112) can be accumulated by the adder of each PE.For example, a PE 112 a (of the column 112) may generate a first scaledinput (from the first input data set), wherein the first scaled inputmay be based on the outputs of the adder. For example, the adder maygenerate a first output partial sum and the PE 112 a may generate afirst scaled input based at least in part on the first output partialsum. The PE 112 a may transmit the first scaled input to a PE 112 b viathe column output bus 106 as a partial sum. The PE 112 b may alsogenerate a second scaled input (from the second input data set) and addthe second scaled input to the partial sum. The updated partial sum,accumulated with the first scaled input and the second scaled input, isthen transmitted to a PE 112 c via the column output bus 106. Thepartial sums are updated and propagated across the column 112, and a PE112 d may generate a sum of the scaled inputs from the four input datasets.

The sum generated by the PE 112 d may correspond to an output data set,and may be fed back to the leftmost PEs after going through anactivation function. Moreover, each PE in the column 112 can alsopropagate the input data sets to other PE columns (e.g., a column 114),which can scale the input data sets with a different set of weights fromthe column 112. Each column of the PEs can perform the arithmeticoperations (multiplications and additions) to generate the output dataelements for other processing elements in parallel. In the example ofFIG. 1A, the systolic array 100A can generate output data elements forfour PEs corresponding to the four columns of the systolic array 100A.

The systolic array 100A may perform convolution computations in multiplewaves. In one embodiment, a wave represents a stream of input dataelements processed while reusing the same weights in the systolic array100A. For example, the respective weights may have been pre-loaded ineach PE in the systolic array 100A, sequentially or in parallel prior tostarting a wave computation. The partial sums generated by the PEs maycorrespond to a single wave. As the PEs of the systolic array 100Aperform arithmetic operations for the convolution computations, dynamicpower dissipated by all the multipliers in the PEs may be significant.This problem may be further exacerbated for a systolic array comprisinga large number of PEs (e.g., several thousands). The arithmeticoperations performed by a PE are further explained with reference toFIG. 2A and FIG. 2B.

As noted above, an input may be reduced to generate a reduced input thatis provided to the systolic array. Further, the input may be reducedinto multiple reduced inputs for multiple single reduced precisioncomputations that are combinable into a higher precision computation.The systolic array may include an aggregator in order to combine partialoutputs into the higher precision output (e.g., a higher precisionoutput relative to the single-pass computation). FIG. 1B illustrates anexample configuration of an eight-PE column 120 within a systolic array100B. The array 100B may be similar to the array 100A of FIG. 1A, butillustratively includes 8 rows and one column. Specifically, as shown inFIG. 1B, an input may be converted into multiple reduced inputs and eachPE may perform a multiply-accumulate operation on each combination ofreduced inputs and provide a partial output partial sum to correspondingadjacent PE. By varying the number of reduced inputs, the number ofpartial output partial sums generated and the number ofmultiply-accumulate operations may be similarly varied. Thus, eachhigher bit-length input may be converted into any number of reducedinputs with lower bit-lengths by the reducer for the systolic array inorder to satisfy the bit-lengths natively supported by the systolicarray.

To facilitate calculation of a total output sum for a column, the column120 in FIG. 1B includes an aggregator 130. The aggregator 130 may belocated within or outside the array 100B. For each pass through thearray for a given input (e.g., for each combination of reduced inputsassociated with a particular input), the aggregator 130 may store andsum the partial outputs. The aggregator 130 may add the partial sumsgenerated for each combination of reduced inputs. The aggregator 130 maycalculate a running sum (e.g., by iteratively adding the partial outputsums for a given set of reduced inputs) for output as the total outputsum. For example, the aggregator 130 may include a partial sum buffer132.

In some embodiments, the systolic array may identify a particular orderto pass the reduced inputs and the reduced weights through the array.For example, the reduced inputs and the reduced weights may be passedfirst through the array in order to retain the accuracy of the numberswith a lower magnitude. Therefore, the reduced inputs with lowermagnitude may be accumulated first in order to retain accuracy. Forexample, the product of a low reduced input data element and a lowreduced weight may be added to the product of a high reduced input dataelement and a low reduced weight (or a low reduced input data elementand a high reduced weight) to generate a first partial output. Further,the first partial output may be added to the product of a low reducedinput data element and a high reduced weight (or a product of the highreduced input data element and a low reduced weight) to generate asecond partial output. Further, the second partial output may be addedto the other of the product of the low reduced input data element andthe high reduced weight or the product of the high reduced input dataelement and the low reduced weight to generate a third partial output.The third partial output may be added to the product of a high reducedinput data element and a high reduced weight to generate a total output.By adding the reduced inputs with the lower magnitude first, theprecision of the reduced inputs may be maintained in order to minimizethe loss of precision of the low reduced inputs when added to the highreduced inputs.

While an aggregator 130 providing pairwise summation is shown in FIG.1B, the aggregator 130 may alternatively implement other aggregationtechniques. In some implementations, the column 120 of the PEs may notinclude an aggregator 130 and may provide an output data set consistingof partial sums for each combination of reduced inputs. In oneimplementation, the column 120 may not include an aggregator 130 and thecolumn 120 may provide multiple partial output data sets. In someembodiments, the multiple output data sets may each correspond to apartial sum generated for each combination of reduced inputs of thecolumn 120. In another implementation, the aggregator 130 may providemore or less output data sets. The aggregator 130 may provide one ormore output data sets each corresponding to one or more partial sums. Insome instances, output of the aggregator 130 may be configurableaccording to a desired use of the array, and may therefore acceptinstructions as to what outputs should be provided. In some instances,the aggregator 130 may provide a combination of the above outputs (e.g.,by providing the four partial sums corresponding to each combination ofreduced inputs, as well as a final sum for the non-reduced input). Insome embodiments, a portion of the aggregation of the partial sums mayoccur within the systolic array. For example, the systolic array may add(using one or more components) a first partial sum and a second partialsum to generate a third partial sum and may add a fourth partial sum anda fifth partial sum to generate a sixth partial sum. Further, thesystolic array may provide the third partial sum and the sixth partialsum for accumulation to the aggregator 130.

FIG. 2A illustrates a PE 00 in a systolic array for neural networkcomputations, according to certain embodiments of the disclosedtechnologies. The PE 00 may be part of a systolic array similar to thesystolic array 100A in FIG. 1A. FIG. 4A and FIG. 4B show additionaldetails of the reducers 225, 227 of FIG. 2A. Some embodiments may bedescribed with reference to neural networks, however, it will beunderstood that certain embodiments may be used in other applications,e.g. pattern recognition, image processing, audio processing, videoprocessing, etc., without deviating from the scope of the technologies.

The systolic array 200 includes reducers 225, 227 and a plurality ofprocessing elements including PE 00 and PE 01. The PE 00 may include oneor more of a data element load generator 202, an input data elementregister 204, a weight register 206, a multiplier 208, an adder 210, askip calculation generator 212, a skip calculation register 214, aselector circuit 216, an input partial sum register 218, a cached weightregister 220, and an operation decoder 256. The PE 00 may receive one ormore of a reduced input data element 222, a reduced weight 224, a zerodata element indicator 226, a zero weight indicator 228, an opcode 230,a weight load 232, and an input partial sum 234 to perform theconvolution computations according to some embodiments.

The PE 00 may be connected to a first reducer 225 and a second reducer227. The first reducer 225 may receive a first input (such as input dataelement 221), and the second reducer 227 may receive a second input(such as weight 223). The first reducer 225 may convert the first inputinto a first reduced input, and the second reducer 227 may convert thesecond input into a second reduced input. The first reducer 225 mayprovide the PE 00 with the reduced input data element 222 (e.g., areduced version of the input data element 221). Further, the secondreducer 227 may provide the PE 00 with the reduced weight 224 (e.g., areduced version of the weight 223). In some embodiments, one or more ofthe first reducer 225 or the second reducer 227 may round the inputand/or the reduced input. The rounding may be based on a rounding methodidentified by the system, a user, etc. (e.g., a user input may specify aparticular rounding method). In other embodiments, one or more of thefirst reducer 225 or the second reducer 227 may reduce a pre-roundedinput (e.g., the pre-rounded input may be rounded by a system local toor remote to the systolic array). Further, the first reducer 225 and thesecond reducer 227 may convert one or more floating-point inputs into areduced representation. The floating-point inputs may includebit-lengths of 32-bits, 64-bits, or any number of bits.

In some embodiments, one or more of the first reducer 225 or the secondreducer 227 may detect when one or both of the input data element 221and the weight 223 exceed a particular bit-length. For example, thefirst reducer 225 may determine if the input data element 221 exceeds22-bits and the second reducer 227 may determine if the weight 223exceeds 22-bits. Further, a user, the system, etc. may provide theparticular bit-length for comparison with the bit-length of the inputdata element 221 and the weight 223. Upon determining that a particularinput (e.g., the input data element 221) exceeds the identifiedbit-length, one or more of the first reducer 225 or the second reducer227 can generate a reduced input (e.g., a reduced input data element222).

In order to reduce the bit-length of the input data element 221 and/orthe weight 223, the first reducer 225 and/or the second reducer 227 canreduce the bit-length of a significand portion of the particular length.The first reducer 225 and/or the second reducer 227 can reduce thebit-length of the significand portion to match the maximum bit-length ofthe significand supported by components of the systolic array (e.g., themultiplier of each processing element). For example, the first reducer225 and/or the second reducer 227 can reduce the bit-length of asignificand portion of the input from 23-bits to 11-bits. In someembodiments, the first reducer 225 and/or the second reducer can expandan exponent portion of the input to a particular format required by themultiplier. For example, the first reducer 225 and/or the second reducer227 can expand the bit-length of the exponent portion of the input from8-bits to 10-bits.

In the event that the significand portion of one or both of the inputdata element 221 and the weight 223 are already reduced, the firstreducer 225 and the second reducer 227 can still extend the number ofbits used to represent the exponent portion of each. Accordingly,subsequent arithmetic circuits such as the multiplier 208 can performcomputations on numbers of a single format (e.g., 22-bit floating-pointformat).

The PE 00 may receive the reduced input data element 222 via a firstinput port. The reduced input data element 222 may be an input data set,or any array of input data elements. The PE 00 may receive one reducedinput data element at a time, in uniform time periods, from the inputdataset. For example, a uniform time period may correspond to a clockcycle. The input data set may be similar to an input feature mapcomprising input feature map elements. As an example, the input data setmay correspond to an input image, an audio clip, a video clip, a textportion, or any other data which may be provided for data processing toidentify a certain pattern or an object. In some instances, the inputdata set may be an intermediate output dataset, which has gone throughan activation function, e.g., ReLu or Sigmoid, as discussed withreference to FIG. 1A. Each reduced input data element 222 may afloating-point data type or any suitable data type. Each reduced inputdata element 222 may include 22-bits, 21-bits, 20-bits, or any suitablenumber of bits. The reduced input data element 222 may be stored in theinput data element register 204 for a period of time.

The PE 00 may receive the reduced weight 224 via a second input port. Insome embodiments, the reduced weight 224 may belong to a set of weightvalues corresponding to a convolution filter. The reduced weight 224 maybe pre-loaded in the PE 00 prior to receiving the reduced input dataelement 222. In some embodiments, the PE 00 may receive one reducedweight value at a time, in uniform time periods, from the set of reducedweight values, to pre-load each PE in a given row with a respectivereduced weight value. The PE may pass the reduced weight value to thenext PE in the respective row until each PE in the given row has beenpre-loaded. Each PE may cache the respective reduced weight value to usefor computations with the reduced input data elements. Each reducedweight 224 may be a floating-point data type or any suitable data type.Each reduced weight 224 may include 22-bits, 21-bits, 20-bits, or anysuitable number of bits. The reduced weight 224 may be stored in acached weight register 220 for a period of time.

The PE 00 may receive the input partial sum 236 for a current operationvia a third input port. In some embodiments, the input partial sum 236can be a 16 bit, 18 bit, 32, bit, 33 bit, 34 bit number or have anynumber of bits.

The PE 00 may receive the zero data element indicator 226 for a currentoperation via a fourth port. The zero data element indicator 226 mayinclude a single bit or multiple bits. The zero data element indicator226 may indicate (or be used to indicate) whether the reduced input dataelement 222 is zero. The zero data element indicator 226 may indicatewhether the input data element 221 is zero. For example, a value of “1”for the zero data element indicator 226 may indicate that the reducedinput data element 222 associated with the zero data element indicator226 is zero, and a value of “0” for the zero data element indicator 226may indicate that the reduced input data element 222 associated with thezero data element indicator 226 is not zero. Further, a “0” maycorrespond to a logical zero or a logical low, and a “1” may correspondto a logical one or a logical high. For example, the logical zero may berepresented by a first range of voltage levels (e.g., 0-2 volts), andthe logical one may be represented by a second range of voltage levels(e.g., 3-5 volts). It will be understood that other implementations torepresent a “0” value and a ‘1” value are possible without deviatingfrom the scope of the disclosed technologies. The zero data elementindicator 226 may be generated by a circuit external to the PE 00, andpassed to all the PEs in the same row sequentially, in the uniform timeperiods.

The PE 00 may receive the zero weight indicator 228 via a fifth port.The zero weight indicator 228 may include a single bit or multiple bits.The zero weight indicator 228 may indicate whether the reduced weight224 associated with the zero weight indicator 228 is zero. The zeroweight indicator 228 may also indicate whether the weight 223 associatedwith the zero weight indicator 228 is zero. For example, a value of “1”for the zero weight indicator 228 may indicate that the reduced weight224 is zero, and a value of “0” for the zero weight indicator 228 mayindicate that the reduced weight 224 is not zero. The zero weightindicator 228 may be generated by a circuit external to the PE 00, andpassed to all the PEs in the same row sequentially along with thereduced weight 224.

The weight load 232 may load the reduced weight 224 into the cachedweight register 220 to provide a cached weight 246. The weight load 232may be asserted to cache the reduced weight 224 for the PE 00 in thecached weight register 220 before the reduced input data element 222 isfed into the array. As the weights are shifted into the array topre-load each PE with a respective weight value, the weight load 232 maybe asserted for each PE at certain time periods in order to pre-loadeach PE with the appropriate weight value.

The operation decoder 256 may decode the opcode 230 to determine anoperation to be executed by the PE 00 for different instructionsrepresented by different opcode values. In some embodiments, a firstopcode value may correspond to an instruction to shift the reducedweights from one PE to another in the systolic array. A second opcodevalue may correspond to an instruction to start the arithmeticcomputations by the PE. For example, once the reduced weights have beenpre-loaded in the systolic arrays, the reduced input data elements maybe read from the memory and the arithmetic computations may be performedas the reduced input data elements pass through the array. A thirdopcode value may correspond to an instruction to execute NOPs. The NOPSmay be used to space two systolic array instructions, or when there areno reduced input data elements to be read from the memory. For example,the NOPs may be used to space the instructions to shift the reducedweights, and the instructions to start the arithmetic computations. Forexample, for a 4×4 array, it may take up to 15 cycles to shift thereduced weights into all the PEs in the array before starting thearithmetic computations so 15 NOP cycles may be needed. The operationdecoder 256 may decode the opcode 230 to generate a NOP 258, and thestart computations signal 260. The operation decoder 256 may provide thestart computations signal 260 to the weight register 206 that isconnected to the multiplier 208 and to the adder 210. The operationdecoder 256 may also provide the start computations signal 260 to themultiplier 208. The opcode 230 may include any suitable number of bits,e.g., two, four, etc. In some embodiments, the operation decoder 256 canalso decode the opcode to determine a data type to provide a data typecontrol signal.

In some embodiments, the reduced input data element 222, the reducedweight 224, the opcode 230, the zero data element indicator 226, and thezero weight indicator 228 may belong to the row input bus 102, asdiscussed with reference to FIG. 1A. In other embodiments, a splitter(not shown) may be used in the PE 00 to split the row input bus 102 intodifferent internal buses to carry the reduced input data element 222,the reduced weight 224, the opcode 230, the zero data element indicator226, and the zero weight indicator 228 within the PE 00. For example,the reduced input data element 222 and the reduced weight 224 may belongto a first row input bus and the opcode 230, the zero data elementindicator 226, and the zero weight indicator 228 may belong to a secondrow input bus.

The data element load generator 202 may generate a data load signal 242that may be used to allow the input data element register 204 to skipstoring of the reduced input data element 222 in certain conditions. Insome embodiments, the reduced input data element 222 may be loaded intothe input data element register 204 when the data load signal 242 isasserted based on the zero data element indicator 226 and the NOP 258.The data load signal 242 may be asserted when the zero data elementindicator 226 corresponding to the reduced input data element 222 is “0”and the opcode 230 does not indicate a NOP (e.g., the NOP 258 is “0”).The data load signal 242 may not be asserted when the zero data elementindicator 226 corresponding to the reduced input data element 222 or theNOP 258 is “1.” The data element load generator 202 may be implementedusing an OR, NOR, NAND, or any suitable circuit.

The input data element register 204 may store the reduced input dataelement 222, or skip storing of the reduced input data element 222 toprovide a stored input data element 244 based on the data load signal242 for a current operation. In some embodiments, the input data elementregister 204 may store a Din input if a load input is “1”, and may holdthe previous value if the load input is “0.” For example, if the dataload signal 242 is “1,” the input data element register 204 may store anew value for the reduced input data element 222, and if the data loadsignal 242 is “0,” the input data element register 204 may skip storingthe new value for the reduced input data element 222. Thus, in someinstances, the input data element register 204 may only store non-zerovalue of the reduced input data element 222. According to certainembodiments, skipping the storing of the new value by the input dataelement register 204 may result in not toggling the stored input dataelement 244 and holding the previous value of the stored input dataelement 244.

The weight register 206 may store the cached weight 246 to provide astored weight value 248 based on the start computations signal 260. Insome embodiments, the weight register 206 may store a Din input if aload input is “1,” and may hold the previous value if the load input is“0.” For example, if the start computations signal 260 is asserted(e.g., the start computations signal 260 is “1”), the cached weight 246may be loaded into the weight register 206, else the weight register 206may hold the previous value. Thus, the reduced weight 224 previouslyloaded into the cached weight register 220 using the weight load 232 maybe shifted into the weight register 206 at the start of the arithmeticcomputations. In some embodiments, the stored weight value 248, onceloaded at the start of the arithmetic computations, remains unchanged asthe input data element is fed into the PE 00, one element at a time, forcomputations corresponding to one or more waves through the systolicarray.

The PE 00 may provide the stored input data element 244 to a PE 01 basedon the data load signal 242 for a current operation. The PE 01 mayreceive the stored input data element 244 via a first port as a reducedinput data element 222. In some embodiments, the input data elementregister 204 may store a Din input if a load input is “1”, and may holdthe previous value if the load input is “0.” The PE 00 may provide thestored weight value 248 to a PE 01 based on a start computations signal260. The PE 01 may receive the stored weight value 248 via a second portas a reduced weight 224. In some embodiments, the weight register 206may store a Din input if a load input is “1,” and may hold the previousvalue if the load input is “0.”

The multiplier 208 may perform a multiplication operation between thestored input data element 244 and the stored weight value 248. Themultiplier 208 may generate a product 250 based on the multiplicationoperation. The multiplier 208 may receive inputs of a fixed bit-length.For example, the multiplier 208 may receive 22-bit floating-pointinputs. Therefore, the reducer can enable the systolic array to receiveinputs of an arbitrary bit-length and provide the multiplier 208 with areduced input of a bit-length supported by the multiplier 208. In someembodiments, the product 250 may be an integer product, a floating-pointproduct, or any other product. Further, the multiplier 208 may generatea product 250 of 8-bits, 16-bits, 18-bits, 32-bits, 34-bits, or anyother number of bits. The multiplier 208 may be implemented using amultiplier circuit. The multiplier 208 may perform floating-pointmultiplication, integer multiplication, or multiplication involving anyother data type. The multiplier 208 may be implemented using a 16-bitmultiplier data path, an 18-bit multiplier data path, a 22-bitmultiplier data path, or a multiplier data path with any number of bits.The multiplier 208 may support at least n-bits operations, wherein n isgreater than or equal to the number of bits in the input (e.g., theinput data element).

The multiplier 208 may contain multiple data paths, for example, asfurther discussed with respect to FIG. 5 . With respect to FIG. 2A, themultiplier 208 may contain separate data paths for computing a sign bit,a significand, and an exponent. It will be understood that thesignificand data path and the exponent data path may include data of anynumber of bits.

The multiplier 208 may provide the product 250 to the adder 210. Theadder 210 may perform an addition operation on the product 250 and thestored input partial sum 236 to provide an addition result 238. Theadder 210 may be implemented using an adder circuit. The adder 210 mayperform floating-point addition, integer addition, or non-integeraddition. The adder 210 may perform addition on inputs with 8-bits,16-bits, 18-bits, 32-bits, 34-bits, or any number of bits. The adder 210may be implemented using a 16-bit adder data path, an 18-bit adder datapath, a 32-bit adder data path, a 34-bit adder data path, or an adderdata path with any number of bits. In one embodiment, the adder 210 isimplemented with given bit-size (e.g., with an adder data path of thegiven bit-size), which may represent a maximum bit size of an expectedinput to the array. In some embodiments, each processing element mayinclude an adder with a larger bit-size and a multiplier with a smallerbit-size as adders of increased bit-sizes may be more cost efficientthan multipliers of the same increased bit-sizes. Therefore, thisdisclose enables a systolic array to support, at reduced precision,larger bit-sizes using lower bit-size multipliers. In anotherembodiment, the adder 210 may be implemented with a smaller bit sizethan a maximum bit size of an expected input to the array. The adder 210may support at least m-bits operations where m is equal to or largerthan the value of the multiplier data path. The adder data path may be asuperset of the multiplier data path.

The multiplier 208 and the adder 210 may provide a fusedmultiply-accumulate operation. The multiplier 208 and the adder 210 maybe integrated together to perform a single step multiply add operation.In some embodiments, no rounding may be performed on the output of themultiplier 208 prior to providing the output to the adder 210. Further,the multiplier 208 may provide an accurate product 250 to the adder 210.In other embodiments, the PE 00 may perform rounding on the output ofthe multiplier 208.

The selector circuit 216 may receive the addition result 238, the inputpartial sum 236, and the stored skip calculation indicator 254. Theselector circuit 216 may select either the addition result 238 or theinput partial sum 236 to provide as an output partial sum 240 via asixth port. In some embodiments, the selector circuit 216 may contain atleast one multiplexer, the multiplexer may select the addition result238 or the input partial sum 236 to be produced. The selector circuit216 may select either the addition result 238 or the input partial sum236, based on the stored skip calculation indicator 254, to provide asan output partial sum 240 via a sixth port. According to someembodiments, when a value of either the reduced input data element 222or the reduced weight 224 for a current operation is zero, or the NOP258 is asserted, the addition result 238 since the product 250 may holda value for the previous operation. In such cases, the stored skipcalculation indicator 254 may allow bypassing the addition result 238,and selecting the input partial sum 236 to provide as the output partialsum 240. For example, when the stored skip calculation indicator 254provides a skip calculation signal of “1”, the input partial sum 236 maybe selected as the output partial sum 240 for a systolic cycle, and whenthe stored skip calculation indicator 254 provides a skip calculationsignal of “0”, either the addition result 238 may be selected as theoutput partial sum 240 for the systolic cycle.

FIG. 2B illustrates the figure shown in FIG. 2A with a shared reducer225 replacing the first reducer 225 and the second reducer 227. Theshared reducer 225 may receive the input data element 221 and the weight223. The shared reducer 225 may also receive the opcode 230. The sharedreducer 225 may perform a selection operation on the input data element221 and the weight 223 based at least in part upon the opcode 230. Insome embodiments, the shared reducer 225 will produce a reduced inputbased at least in part upon the opcode 230. For example, when the opcode230 is a particular value, the shared reducer 225 may reduce the weight223 and provide the reduced weight 224 to the PE 00. Further, when theopcode 230 provides some other set value, the shared reducer 225 mayreduce the input data element 221 and provide the reduced input dataelement 222 to the PE 00. Therefore, the shared reducer 225 can reducethe bit-length of the significand portion of both the input data element221 and the weight 223 to match the maximum bit-length of thesignificand supported by components of the systolic array (e.g., themultiplier of each processing element). In some embodiments, the sharedreducer 225 may receive multiple input data elements and/or multipleweights and produce multiple reduced input data elements and/or multiplereduced weights. For example, the shared reducer 225 can produce anynumber of reduced input data elements (e.g., four) and/or any number ofreduced weights (e.g., four).

The shared reducer 225 may use a multiplexer to select between the inputdata element 221 and the weight 223. In some embodiments, the reducedinput data element 222 and the reduced weight 224 may be delivered tothe PE 00 on separate buses. In other embodiments, the reduced inputdata element 222 and the reduced weight 224 may be delivered on the samebus. Further, the shared reducer 225 may reduce both the input dataelement 221 and the weight 223 in the same clock cycle and provide thereduced input data element 222 and the reduced weight 224 to the PE 00.In some embodiments, the shared reducer 225 may reduce the weight 223and provide the reduced weight 224 to the PE 00 during a clock cycle.The shared reducer 225 may then reduce the input data element 221 andprovide the reduced input data element 222 to the PE 00 during a secondclock cycle.

FIG. 3 illustrates an apparatus 300 including zero detector circuits forreduced input data elements and reduced weights entering a systolicarray for neural network computations, according to certain embodimentsof the disclosed technologies.

The apparatus 300 may include a two-dimensional systolic array 302comprising PEs arranged into rows and columns. The systolic array 302may be similar to the systolic array 100A in FIG. 1A. A first row of thesystolic array 302 may include PE 00, PE 01, PE 02, . . . , PE 0y, asecond row of the systolic array 302 may include PE 10, PE 11, PE 12, .. . , PE 1y, a third row of the systolic array 302 may include PE 20, PE21, PE 22, . . . , PE 2y, and an Xth row of the systolic array 302 mayinclude PE x0, PE x1, PE x2, . . . , PE xy. The x and y may includepositive integers, e.g., 32, 64, 128, or any suitable number. Each PE ofthe systolic array 302 may be similar to the PE 01, and include means toperform arithmetic computations on reduced inputs using power efficientmethods, as discussed with reference to FIG. 2A, FIG. 2B.

In certain embodiments, a first (e.g., leftmost) PE in each row of thesystolic array 302 may be coupled to a respective zero input datadetector circuit to detect a zero value on an input data element, and arespective zero weight detector circuit to detect a zero value on aweight value entering the systolic array 302. For example, the PE 00 inthe first row may be coupled to a first zero input data detector 306 aand a first zero weight detector 308 a, the PE 10 in the second row maybe coupled to a second zero input data detector 306 b and a second zeroweight detector 308 b, the PE 20 in the third row may be coupled to athird zero input data detector 306 c and a third zero weight detector308 c, and the PE x0 in the Xth row may be coupled to an Xth zero inputdata detector 306 x and an Xth zero weight detector 308 x. The firstzero input data detector 306 a, the second zero input data detector 306b, the third zero input data detector 306 c, . . . , and the Xth zeroinput data detector 306 x may detect a zero value on a respectivereduced input data element in an input dataset0, an input dataset1, aninput dataset2, . . . , and an input datasetx respectively. Similarly,the first zero weight detector 308 a, the second zero weight detector308 b, the third zero weight detector 308 c, . . . , and the Xth zeroweight detector 308 x may detect a zero value on a respective reducedweight value in a filter0, a filter1, a filter2, . . . , and a filterxrespectively.

Each zero input data detector and each zero weight detector in each rowof the systolic array 302 may be coupled to a respective reducer toreceive a reduced input. Each zero input data detector may receive areduced input data element and each zero weight detector may receive areduced weight. For example, the first zero input data detector 306 amay be coupled to a first reducer 307 a and the first zero weightdetector 308 a may be coupled to a second reducer 309 a, the second zeroinput data detector 306 b may be coupled to a third reducer 307 b andthe second zero weight detector 308 b may be coupled to a fourth reducer309 b, the third zero input data detector 306 c may be coupled to afifth reducer 307 c and the third zero weight detector 308 c may becoupled to a sixth reducer 309 c, and the Xth zero input data detector306 x may be coupled to an Xth reducer 307 x and the Xth zero weightdetector 308 x may be coupled to an Yth reducer 309 x.

The reducers 307 a-307 x and 309 a-309 x may be implemented as aseparate entity external to the systolic array 302. For example, thereducers 307 a-307 x and 309 a-309 x may be part of a circuit separatefrom the systolic array. In some embodiments, the circuit and thesystolic array 302 may be part of a computing engine, which may performarithmetic computations for the convolution operations. In otherembodiments, the reducers 307 a-307 x and 309 a-309 x may be implementedas part of the systolic array 302.

In some embodiments, the first reducer 307 a and the second reducer 309a may be a first shared reducer and the third reducer 307 b and thefourth reducer 309 b may be a second shared reducer and the fifthreducer 307 c and the sixth reducer 309 c may be a third shared reducerand the Xth reducer 307 x and the Yth reducer 309 x may be an Xth sharedreducer. Each shared reducer may provide a reduced input data elementand a reduced weight. In some embodiments, each shared reducer maycontain one output bus and may select a reduced input to produce. Inother embodiments, each shared reducer may contain multiple output busesand may output a reduced input data element and a reduced weight.

The zero input data detectors 306 a-306 x and/or zero weight detectors308 a-308 x can be arranged before the respective reducers 307 a-307 x,309 a-309 x such that a zero input can be detected, and if the zeroinput is detected, then the respective reducer(s) 307 a-307 x, 309 a-309x can be non-operational to conserve power. In some embodiments, boththe zero input data detectors 306 a-306 x and respective reducers 307a-307 x can receive the input datasets and operate in parallel insteadof sequentially. Further, both the zero weight detectors 308 a-308 x andthe respective reducers 309 a-309 x can receive the filters and operatein parallel instead of sequentially.

Each of the input dataset0, the input dataset1, the input dataset2, . .. , and the input datasetx may belong to an image, a text, a video clip,an audio clip, or another type of data set which may need to beprocessed by a neural network processor for convolution computations.

In some instances, the input dataset0, the input dataset1, the inputdataset2, . . . , and the input datasetx may be associated with outputdataset0, output dataset1, output dataset2, . . . , output datasetygenerated by an intermediate layer of the convolution operation. Forexample, the output dataset0, output dataset1, output dataset2, . . . ,output datasety may go through activation functions and be fed back tothe systolic array 302 as the input dataset0, the input dataset1, theinput dataset2, . . . , and the input datasetx. The filter0, thefilter1, the filter2, . . . , and the filterx may include different setsof weight values to convolve with the input dataset0, the inputdataset1, the input dataset2, . . . , and the input datasetx. The weightvalues in the filter0, the filter1, the filter2, . . . , and the filterxmay be pre-determined using supervised learning, non-supervisedlearning, or any suitable method of determining convolution filters.

Each zero input data detector for the respective row may detect whethera reduced input data element from the input dataset entering therespective row is “0” and generate a corresponding zero input dataindicator for that reduced input data element. Further, each zero inputdata detector for the respective row may also detect whether an inputdata element from the input dataset entering the respective reducer is“0” and generate a corresponding zero input data indicator for thatinput data element. The corresponding zero data element indicator may bepassed into the first PE of the respective row along with the reducedinput data element. For example, the PE 00 may be the first PE of thefirst row in the systolic array 302. The PE 00 may receive reduced inputdata elements from the input dataset0 prior to other PEs in the firstrow (e.g., PE 01, PE 02, . . . , PE 0y). In some embodiments, onereduced input data element at a time may be fed sequentially, in uniformtime periods, from the input dataset0 to the PE 00. The first zero inputdata detector 306 a may generate the zero data element indicator 226 ineach of the uniform time periods (e.g. clock cycles) for each input dataelement from the input dataset0. The zero data element indicator 226 maybe fed to the PE 00 sequentially, in uniform time periods, along witheach reduced input data element. The PE 00 may or may not store thereduced input data element 222 based on the value of the respective dataload signal 242. In some embodiments, the first zero input data detector306 a may include a comparator to compare the incoming reduced inputdata element with a zero to assert (e.g., set to “1”) or de-assert(e.g., set to “0”) the zero data element indicator 226 based on thevalue of the incoming reduced input data element. For example, thecomparator may be implemented using an OR, XOR, NAND, or any suitablecircuit.

Each zero weight detector for the respective row may detect whether areduced weight from a set of reduced weights entering the respective rowis zero and generate a corresponding zero weight indicator for thereduced weight. Further, each zero weight detector may also detectwhether a weight from a set of filters entering the respective reducersis zero and generate a corresponding zero weight indicator for thatweight. For example, the first zero weight detector 308 a may detectwhether a reduced weight from the filter0 (e.g., the reduced weight 224)includes a zero value and generate the zero weight indicator 228 for thereduced weight. In some embodiments, the first zero weight detector 308a may include a comparator to compare the reduced weight with a zero toassert (e.g., set to “1”) or de-assert (e.g., set to “0”) the zeroweight indicator 228. For example, the comparator may be implementedusing an OR, XOR, NAND, or any suitable circuit. In one embodiment, areduced weight, one at a time, may be fed sequentially, in uniform timeperiods, from the filter0 to the PE 00 for pre-loading the respectivereduced weights to the PE 00 to the PE 0y prior to starting thearithmetic computations. The first zero weight detector 308 a maygenerate a corresponding zero weight indicator for each of those reducedweights which may be fed to the PE 00 sequentially, in uniform timeperiods, along with the corresponding reduced weight. The PE 00 may passthe respective reduced weight and the corresponding zero weightindicators sequentially to the next neighboring PE until all the PEs inthe first row have been preloaded with the respective reduced weightsand the corresponding zero weight indicators. The respective reducedweights and the corresponding zero weight indicator may be cached ineach PE before the respective reduced input data elements are fed toeach row in the systolic array 302.

The second zero input data detector 306 b, the third zero input datadetector 306 c, . . . , and the Xth zero input data detector 306 x maybe similar to the first zero input data detector 306 a, and may generatea respective zero data element indicator, similar to the zero dataelement indicator 226, to provide to the PE 10, PE 20, . . . , and PEx0, sequentially, in the uniform time periods, for power optimization.The respective zero data element indicator generated for each row may bereceived by a respective first PE in each row via the respective rowinput bus 102, and propagated, sequentially, in the uniform timeperiods, by the first PE to all the PEs in the given row. The secondzero weight detector 308 b, the third zero weight detector 308 c, . . ., and the Xth zero weight detector 308 x may be similar to the firstzero weight detector 308 a, and may generate a respective zero weightindicator, similar to the zero weight indicator 228, to provide to thePE 10, PE 20, . . . , and PE x0, sequentially, to pre-load each PE inthe respective row along with the respective weight value prior tostarting the arithmetic computations.

In some embodiments, the zero input data detectors 306 a-306 x, and thezero weight detectors 308 a-308 x may be implemented as a separateentity external to the systolic array 302. For example, the zero inputdata detectors 306 a-306 x, and the zero weight detectors 308 a-308 xmay be part of a circuit 304. In other embodiments, the circuit 304 andthe systolic array 302 may be part of a computing engine, which mayperform arithmetic computations for the convolution operations. Someembodiments of the disclosed technologies can provide reduced gate countand dynamic power consumption by detecting zeros on the input dataelements and the weights entering a respective first PE in each row ofthe systolic array, and passing the zero indicators to all the PEs inthe array as compared to using respective zero detectors within each PEin the systolic array 302.

Note that FIG. 3 only shows the respective zero data element indicatorand the zero weight indicator entering the first PE in each row of thesystolic array 302 for ease of illustration, however it will beunderstood that each PE in the respective row of the systolic array 302may also receive the respective reduced input data element and therespective reduced weight along with some control signals (e.g., opcode230, weight load 232, data type, etc.), which may be propagated from theleft to the right of the systolic array 302 for each row.

FIG. 4A shows an example reduction system 400A (e.g., a 32-bitfloating-point (“FP32”) reduction system) according to an exampleimplementation. The reduction system 400A includes a multiplexer 402, arounding identifier, and a reducer 405. The reducer 405 may reduce inputof an arbitrary bit-length to the maximum bit-length supported byelements of a systolic array during a single-pass computation. Forexample, the reducer 405 may reduce input to a 22-bit input where22-bits is the maximum bit-length supported by a multiplier of thesystolic array. The reducer 405 can include an exponent expander 406, arounder 408, and a trailing bit reducer 410. In some embodiments, thereducer 405 may include the exponent expander 406. In other embodiments,the reducer 405 may not include the exponent expander 406. For example,the reducer 405 may not expand the exponent of an input to generate thereduced input. In some embodiments, the multiplexer 402 may be separatefrom the reducer 405. In other embodiments, the reducer 405 may includethe multiplexer 402. As previously discussed, the reducer 405 processesan original number 401A to result in a reduced number 403A.

The reduction system 400A may receive one or more numbers to be reduced.The one or more numbers may include one or more of an input data element221 and/or a weight 223. For example, the reduction system 400A canreceive a FP32 weight and an FP32 input data element. In someembodiments, the reduction system 400A may receive the input dataelement 221 or the weight 223 without a multiplexer.

The multiplexer 402 may receive the one or more numbers received by thereduction system 400A. The multiplexer 402 may also receive an opcode230 or other indicator of whether a weight or input data element shouldbe selected. The multiplexer 402 may decode the opcode 230 to select anumber to be operated on by the reduction system 400A. The multiplexer402 may output a different number for the reduction operation based onthe value of the opcode 230. In some embodiments, a first opcode valuemay correspond to an instruction to output the weight 223 as themultiplexer output 420 and a second opcode value may correspond to aninstruction to output the input data element 221 as the multiplexeroutput 420. For example, once the input data element 221 and the weight223 have been provided to the reduction system 400A, the multiplexer 402may output the input data element 221 and, at a later time, the weight223, based at least in part on the opcode 230.

In the example of FIG. 4A, the original number 401A is an FP32 numberwith a sign bit portion, an exponent bit portion, and a significand bitportion. It will be understood that the original number 401A can be anyarbitrary bit-length number with any exponent bit-length and/orsignificand bit-length. The FP32 format of the original number 401includes a 1-bit sign, an 8-bit exponent, and a 23-bit significand. Insome embodiments, the original number 401A may include more, less, ordifferent bits. Further, the original number 401A may include more,less, or different bits for the sign bit portion, the exponent bitportion, and/or the significand bit portion.

The exponent expander 406 may receive the 8-bit exponent 428 from theoriginal number 401A. The exponent expander 406 may increase a quantityof bits representing the exponent 428 from 8 bits to 10 bits. In someembodiments, the exponent expander 406 may add 1, 2, 3, or any number ofbits to the exponent 428. The added quantity of bits can be sufficientto represent the number in a format expected by the PE (e.g., the PE mayexpect a 10-bit exponent). In other embodiments, the exponent expandermay not add any bits to the exponent 428. For example, the exponentexpander 406 (or another component) may determine that a sufficient(e.g., adequate) quantity of bits are included in the exponent 428 andmay not expand the exponent 428.

The exponent expander 406 may expand the exponent 428 and retain thevalue of the exponent 428. The exponent expander 406 may expand theexponent using range translation by copying the most significant bit,appending a second, inverted, copy of the most significant bit, andappending the other bits of the exponent 428 to the end of the expandedexponent 434. For example, if the exponent 428 has a value of“10101010”, the exponent expander 406 may copy the most significant bit“1”, invert the most significant bit once “0”, and append the finalseven bits “0101010” such that the expanded exponent 434 is “100101010”.In some embodiments, the expand expander 406 may perform a differentoperation if the exponent begins with a leading zero. Further, theexponent expander 406 may expand the exponent using range translation bycopying the most significant bit, appending a second copy of the mostsignificant bit, and appending the other bits of the exponent 428 to theend of the expanded exponent 434. For example, if the exponent 428 is“00000000,” the exponent expander 406 may expand the exponent 428 suchthat the expanded exponent 434 is “000000000.” In some embodiments, theexponent expander 406 might add the extra bits of data to any locationof the exponent field depending on the endian format and signed orunsigned representation of the exponent. Therefore, the exponentexpander 406 can expand the exponent 428 to generate the expandedexponent 434.

The exponent expander 406 may provide the expanded version of theexponent 434 as the 10-bit expanded exponent field of the reduced number403A.

The reducer 405 may further receive the rounding identifier 404. Therounding identifier 404 may identify a type of rounding to be performedby the reducer 405. For example, the rounding identifier 404 mayidentify a rounding method such as stochastic rounding, rounding tonearest even, rounding to zero, rounding down, rounding up, or any otherrounding method. Stochastic rounding may include randomly rounding tothe next larger or smaller number. For example, stochastic rounding mayinclude a 50% probability of rounding down and a 50% probability ofrounding up. Further, in stochastic rounding, the probability ofrounding up or rounding down may be based on the relative position ofthe number to be rounded. For example, a number x between y and z mayhave a first probability of rounding up to z equal to (x−y)/(z−y) and asecond probability of rounding down to y equal to (z−x)/(z−y) where yand z can be any numbers and x can be any number between y and z.Rounding to the nearest even may include rounding to the nearest evennumber with a particular number of bits, rounding to zero may includerounding a particular number of bits to zero, rounding up may includerounding a particular number of bits up, and rounding down may includerounding a particular number of bits down. The rounding identifier 404may be provided by a user (e.g., via a user interface), another system,etc. Further, the rounding identifier 404 may be a custom roundingidentifier or a default rounding identifier.

The reducer 405 may contain a rounder 408 to round the significand 430.The rounder 408 may perform rounding based on the rounding methodidentified by the rounding identifier 404. For example, the roundingmethod may be stochastic rounding, rounding to nearest even, rounding tozero, rounding down, rounding up, or any other rounding method. Therounder 408 may perform the rounding based on any bit of thesignificand. Further, the rounder 408 may determine a number of bits tobe reduced by the trailing bit reducer 410 (e.g., a number of bits to bezeroed) and may initiate the rounding at the bit immediately prior tothe bits to be reduced. Further, the rounder 408 can round the bits tobe reduced by the trailing bit reducer 410. For example, if thesignificand 430 includes bits “1110111” and the trailing bit reducer 410determines that the trailing bit reducer 410 will reduce the threetrailing bits (e.g., the first three bits reading from the left toright), the rounder 408 may perform rounding based on the “0” inposition 4. Further, if the rounder 408 determines to perform roundingto zero, the rounder 408 may produce a rounded significand 432“1110000,” if the rounder 408 determines to perform rounding up, therounder 408 may produce a rounded significand 432 “1111000,” etc. Insome embodiments, the rounder 408 may be located logically after thetrailing bit reducer 410 and the rounder 408 may round a reducedsignificand.

The reducer 405 may further contain the trailing bit reducer 410 toreduce the bit representation of the rounded significand 432. Thetrailing bit reducer 410 may receive the rounded significand 432 asinput. The trailing bit reducer 410 may identify a number of bits toreduce from the rounded significand 432. The number of bits to reducemay be based on a difference between the bit-length of the roundedsignificand 432 and a maximum single-pass computational bit-lengthsupported by elements of the systolic array. Further, the number of bitsmay be based on a user input or system input (e.g., an input identifyinga maximum number of bits supported). The number of bits may be trailingbits of the rounded significand 432 (e.g., a number of rightmost bits orthe least significant bits). For example, if the trailing bit reducer410 determines 3 bits should be reduced from the rounded significand432, the trailing bit reducer 410 may identify the 3 bits from right toleft in the rounded significand 432. Further, the bits may correspond topositions 0, 1, and 2 within the original number 401A. The trailing bitreducer 410 may identify the bits and zero the bits (e.g., reduce,eliminate, push to logical zero). In the example of FIG. 4A, thetrailing bit reducer 410 identifies that 12 bits should be reduced fromthe rounded significand 432 and zeros the trailing 12 bits of therounded significand 432. By reducing the bit representation of therounded significand 432, the trailing bit reducer 410 can generate areduced significand 436 that includes only the non-reduced (non-zeroed)bits of the significand 430.

The trailing bit reducer 410 may provide the reduced significand 436 asthe 11-bit rounded significand of the reduced number 403A.

The reduced number 403A may be a second bit-length wherein the secondbit-length is any number of bits smaller than the first bit-length. Insome embodiments, the second bit-length may be the maximum bit-lengthsupported by elements of the systolic array. It will be understood thatthe reduced number 403A can be any arbitrary bit-length number with anyexponent bit-length and/or significand bit-length. In the example ofFIG. 4A, the reduced number 403A may be an 22-bit floating-point numberwith a sign bit portion, an exponent bit portion, and a significand bitportion and the original number 401A may be a 32-bit floating-pointnumber. The reduced number 403A may contain a 1-bit sign (e.g., the sign426), a 10-bit exponent (e.g., the expanded exponent 434), and an 11-bitsignificand (e.g., the reduced significand 436). The reduction system400A may provide the reduced number 403A as a reduced output 421. Thereduced output 421 may be a reduced input data element 222, a reducedweight 224, or any other reduced number.

FIG. 4B shows an example reduction system 400B (e.g., a 32-bitfloating-point (“FP32”) reduction system) according to an exampleimplementation. The reduction system 400B may include a reducer 405 thatmay reduce input of an arbitrary bit-length to the maximum bit-lengthsupported by elements of a systolic array during a single-passcomputation. For example, the reducer 405 may reduce input to a 22-bitinput where 22-bits is the maximum bit-length supported by a multiplierof the systolic array. The reduction system 400B includes componentssimilar to the reduction system 400A except that in FIG. 4B an originalnumber 401B is rounded by a system prior to provision to the reductionsystem 400B.

In the example of FIG. 4B, the original number 401B may be a FP32 numberwith a sign bit portion, an exponent bit portion, and a significand bitportion. It will be understood that the original number 401B can be anyarbitrary bit-length number with any exponent bit-length and/orsignificand bit-length The FP32 format of the original number 401Bincludes a 1-bit sign, an 8-bit exponent, and a 23-bit roundedsignificand. In some embodiments, the original number 401B can includeany number of bits or be associated with any other bit format. The23-bit rounded significand may be rounded by a system external orinternal to the reduction system 400B.

The reducer 405 may further contain the trailing bit reducer 410 toreduce the rounded significand 450. The trailing bit reducer 410 mayreceive the rounded significand 432 as input and reduce the quantity ofbits representing the rounded significand 450 (e.g., from 23-bits to11-bits). The trailing bit reducer 410 can generate a reducedsignificand 452 that includes only the non-reduced (non-zeroed) bits ofthe rounded significand 450. Further, the trailing bit reducer 410 mayprovide the reduced significand 452 as the 11-bit rounded significand ofthe reduced number 403B.

In some embodiments, the reduction system 400B may not receive therounding identifier 404. For example, the rounding identifier 404 may beprovided to the system rounding generating the rounded significand 450in order to identify a rounding method. The reduction system 400B mayprovide the reduced number 403B as a reduced output 441. The reducedoutput 441 may be a reduced input data element 222, a reduced weight224, or any other reduced number.

FIG. 4C shows an example reduction system 400C (e.g., a 32-bitfloating-point (“FP32”) reduction system) according to an exampleimplementation. The reduction system 400C may include a reducer 405 thatmay reduce input of an arbitrary bit-length to multiple reduced inputswith a maximum bit-length supported by elements of a systolic arrayduring a single-pass computation. For example, the reducer 405 mayreduce input to a 21-bit input where 21-bits is the maximum bit-lengthsupported by a multiplier of the systolic array. The reduction system400C includes components similar to the reduction system 400A and 400Bexcept that in FIG. 4C an original number 401C is converted intomultiple reduced inputs by the reducer 405.

In the example of FIG. 4C, the original number 401C may be a FP32 numberwith a sign bit portion, an exponent bit portion, and a significand bitportion. It will be understood that the original number 401C can be anyarbitrary bit-length number with any exponent bit-length and/orsignificand bit-length. The FP32 format of the original number 401Cincludes a 1-bit sign, an 8-bit exponent, and a 23-bit roundedsignificand. In some embodiments, the original number 401C can includeany number of bits or be associated with any other bit format.

The original number 401C as an input 454 may be provided to the formatdetector 456 for normal and/or denormal detection. For example, theformat detector 456 may be a denormal detector and/or a normal detector.The format detector 456 may detect whether the input 454 is normal ordenormal based at least in part on at least one of the value of the1-bit sign, the value of the 8-bit exponent, or the value of the 23-bitsignificand. For example, the format detector 456 may detect a denormalnumber when the 8-bit exponent contains zeros in each bit and thesignificand is nonzero. The format detector 456 may provide an enablesignal 458 to the normalizer 455 based at least in part on the detectionof a normal number. For example, if the format detector 456 detects thatthe input 454 is normal, the format detector 456 may provide a firstvalue to the normalizer 455. If the format detector 456 detects that theinput 454 is denormal, the format detector 456 may provide a secondvalue to the normalizer 455. In some implementations, the first numbermay be a 1 and the second number may be a 0. The detection of a normalnumber may correspond to a logical high and the detection of a denormalnumber may correspond to a logical zero. In some embodiments, the formatdetector 456 may detect a normal number by zeroing out the significand450 (e.g., replacing the significand 450 with zeros) and subtracting theoriginal number 401C with the reduced significand 451 from the originalnumber 401C with the zeroed significand to generate a normal identifier.Further, the normal identifier may contain the implied leading bit ifthe original number 401C is normal and may equal zero if the originalnumber 401C is denormal.

The reducer 405 may provide the 1-bit sign as a 1-bit sign of thereduced number 403C and the reduced number 403D.

The reducer 405 may further contain the trailing bit reducer 410 and theleading bit reducer 453 to reduce the significand 450. The trailing bitreducer 410 and the leading bit reducer 453 may receive the significand432 as input and reduce the quantity of bits representing thesignificand 450 (e.g., from 23-bits to 11-bits). The trailing bitreducer 410 can generate a reduced significand 452 that includes onlythe non-reduced (non-zeroed) bits of the significand 450 by removingtrailing (or low) bits of the significand 450. The leading bit reducer453 can generate a reduced significand 451 that includes only thenon-reduced (non-zeroed) bits of the significand 450 by removing highbits of the significand 450. Further, the trailing bit reducer 410 mayprovide the reduced significand 452 as the 11-bit reduced significand ofthe reduced number 403C and the leading bit reducer 453 may provide thereduced significand 451 as the input to the normalizer 455.

As discussed above, the reducer 405 may further contain the exponentexpander 406A and 406B to expand the exponent 428. The exponent expander406A can generate an expanded exponent 434 and may provide the expandedexponent 434 as an exponent of the reduced number 403C and the expandedexpander 406B may provide the expanded exponent 433 as the input to theexponent adjuster 435.

The reducer 405 may contain the normalizer 455 (e.g., a shifter). Thenormalizer 455 may be enabled based at least in part on the enablesignal 458 received from the format detector 456. The normalizer 455 mayreceive the reduced significand 451 from the leading bit reducer 453.The normalizer 455 may shift the reduced significand 451 based at leastin part upon the number of leading zeros of the reduced significand 451(as detected by the normalizer 455). The normalizer 455 may furthershift the reduced significand 451 such that the first non-zero number isshifted out of the reduced significand 451 and represented with animplied bit. The normalizer 455 may shift the reduced significand 451 byadding bits containing logical lows or zeros to the right or end of thereduced significand 451. The normalizer 455 may produce a shiftedsignificand 452, wherein the shifted significand 452 may be the samenumber of bits as the reduced significand 451. For example, if thereduced significand 451 is 00001100000, then the normalizer 455 cancount four zeros and further adjust the shift count to five, and thenormalizer 455 may shift the reduced significand 451 a total of fivetimes and produce a shifted significand 452 of 10000000000. Thenormalizer 455 may then provide the shifted significand 452 as thesignificand portion of the reduced number 403D. In the event that theformat detector 456 does not identify the original number 401C is anormal number (e.g., the original number 401C is a denormal number), thenormalizer 455 can provide the reduced significand 451 as thesignificand portion of the reduced number 403D. In some embodiments, ifthe format detector 456 determines the original number 401C is normal,the reducer 405 may calculate a zeroed number by zeroing the significandof the original number 401C. Further, the reducer 405 may generate thesignificand of the reduced number 403D by subtracting the reducedsignificand from the zeroed number. In other embodiments, the reducednumber 403D may be determined by subtracting the reduced number 403Cfrom the original number 401C.

The exponent expander 406B may provide the expanded version of theexponent 433 to the exponent adjuster 435 (e.g., a subtractor) based atleast in part on the enable signal 458 when a normal format for thefirst input is detected by the format detector 456 and a signal 437 fromthe normalizer 455 identifying the renormalized significand 452. Theexponent adjuster 435 may receive the expanded exponent 433 from theexponent expander 406B and a number of leading zeros from the normalizer455. The number of leading zeros may identify the number of leadingzeros removed by the normalizer 455 in order to renormalize the reducedsignificand 451. The exponent adjuster 435 may subtract a value from theexpanded exponent 433 based at least in part on the leading zeros outputby the normalizer 455. Therefore, the exponent adjuster 435 maycompensate the exponent value for the shift of the significand. Forexample, if the leading zeros output is equal to 5 and the expandedexponent is equal to 000011111 or 31, the exponent adjuster 435 maysubtract 5 from 000011111 or 31, such that the adjusted exponent 439 isequal to 000011010 or 26. The exponent adjuster 435 may provide theadjusted exponent 439 as the 9-bit expanded exponent field of thereduced number 403D. Otherwise, the expanded version of the exponent 433can be stored as the 9-bit expanded exponent field of the reduced number403D. In some embodiments, the exponent expander 406B may expand theexponent 433 prior to the normalizer 455 normalizing the reducedsignificand 451. In other embodiments, the exponent expander 406B mayexpand the exponent 433 after or in parallel with the normalizer 455normalizing the reduced significand 451.

The reduction system 400C may provide the reduced number 403C and thereduced number 403D as reduced inputs 457 and 459 for the originalnumber 401C. The reduced inputs 457 and 459 may be reduced input dataelements 222, reduced weights 224, or any other reduced numbers.

FIG. 5 shows an example multiply accumulate datapath 500. The exampledatapath 500 may be implemented as the multiplier 208 and the adder 210discussed with respect to FIG. 2A and FIG. 2B. As shown in FIG. 5 , themultiplier 208 may receive a reduced input data element 222 and areduced weight 224 and provide a multiplication product to the adder210. The adder 210 may receive the multiplication product and the inputpartial sum 234 and provide an addition result 238. By converting inputsinto reduced representation before presenting inputs to the multiplier208, the multiplier 208 can omit support for numbers with largerbit-lengths (e.g., 32-bits), instead the multiplier 208 can supportnumbers with the reduced bit-lengths (e.g., 22-bits). Therefore, thesystolic array can retain the performance offered by receiving inputs ofshorter bit-lengths by receiving inputs of arbitrary bit-lengths andadjusting the input to a particular bit-length (e.g., the maximumbit-length supported by the processing elements of the systolic array).

The reduced input data element 222 may be a 22-bit number. In someembodiments, the reduced input data element 222 may have any bit-lengthand/or be any number of bits. Further, the reduced input data element222 may be a floating-point number. In some embodiments, the reducedinput data element 222 may be a brain floating-point number. Further,the reduced input data element 222 may be a number of any data type. Thereduced input data element 222 may consist of a sign bit field, anexponent field, and a significand field. The multiplier 208 can supportreduced input data elements of different types. For example, the reducedinput data element 222 may contain a 1-bit sign, a 10-bit exponent, andan 11-bit significand. Further, the reduced input data element 222 maycontain a 1-bit sign, an 8-bit exponent, and an 11-bit significand. Themultiplier 208 may support both of these types of reduced input dataelements. In some embodiments, the reduced input data element 222 maycontain an x-bit sign, a y-bit exponent, and a z-bit significand wherex, y, and z may be any number. The reduced input data element 222 may beprovided to the multiplier 208 via a first sign data path 511, a firstexponent data path 521, and a first significand data path 531.

The reduced weight 224 may be a 22-bit number. In some embodiments, thereduced weight 224 may have any bit-length and/or be any number of bits.Further, the reduced weight 224 may be a floating-point number. In someembodiments, the reduced weight 224 may be a brain floating-pointnumber. Further, the reduced weight 224 may be any data type. Thereduced weight 224 may consist of a sign bit path, an exponent bit path,and a significand bit path. For example, the reduced weight 224 maycontain a 1-bit sign, a 10-bit exponent, and an 11-bit significand.Further, the reduced weight 224 may contain a 1-bit sign, an 8-bitexponent, and a 10-bit significand. In some embodiments, the reducedinput data element 222 may contain an x-bit sign, a y-bit exponent, anda z-bit significand where x, y, and z may be any number. The reducedweight 224 may be provided to the multiplier 208 via a second sign datapath 512, a second exponent data path 522, and a second significand datapath 532.

The multiplier 208 may contain a sign data path, an exponent data path,and a significand data path. The multiplier 208 may receive the firstsign data path 511, the first exponent data path 521, and the firstsignificand data path 531 from the reduced input data element 222. Themultiplier 208 may receive the second sign data path 512, the secondexponent data path 522, and the second significand data path 532 fromthe reduced weight 224. In some embodiments, the multiplier 208 may alsoreceive a data type control signal. The multiplier 208 may performmultiplication operations on the received inputs.

The sign data path of the multiplier 208 may receive the first sign datapath 511 and the second sign data path 512. The sign data path mayoutput a partial sign data path 513 based at least in part on the firstsign data path 511 and the second sign data path 512. In someembodiments, the sign data path can be implemented as an exclusive or(XOR) function. The sign data path may provide the partial sign datapath 513 to the adder 210.

The exponent data path of the multiplier 208 may receive the firstexponent data path 521 and the second exponent data path 522. Theexponent data path of the multiplier 208 may contain an adder 526. Insome embodiments, the exponent data path of the multiplier 208 mayinclude a mapper to adjust the output of the multiplier 208 into aformat expected by one or more components of the systolic array (e.g.,an adder separate from the adder 526). For example, an adder of thesystolic array may expect (e.g., operate on) an input with an 11-bitexponent. Further, the mapper may receive the first exponent data path521 and the second exponent data path 522 and perform a mappingoperation to add one or more bits to the exponent of each of the reducedinput data element 222 and the reduced weight 224

The adder 526 may receive the mapped or unmapped versions of the firstexponent data path 521 and the second exponent data path 522. The adder526 may perform addition on the two values received from the firstexponent data path 521 and the second exponent data path 522. The adder526 can also receive shift/carry information (not shown) from thesignificand data path. The adder 526 may provide a partial exponent datapath 523 based at least in part on the addition performed on the twovalues. The partial exponent data path 523 can be 10 bits or other rangesufficient to accommodate the exponent sum without overflow.

The significand data path of the multiplier 208 may receive the firstsignificand data path 531 and the second significand data path 532. Thesignificand data path of the multiplier 208 may contain a binarymultiplier 534 and a format adjuster 536. The binary multiplier 534 maymultiply the value of the first significand data path 531 by the valueof the second significand data path 532. The binary multiplier 534 maygenerate a multiplier product based on the multiplication operation. Insome embodiments, the product may be an integer product, afloating-point product, or any other product. Further, the binarymultiplier 534 may generate a product of 8-bits, 16-bits, 32-bits, orany other number of bits. The product may have a bit-length of a maximumbit-length supported by the elements of the systolic array during asingle-pass computation. Therefore, the systolic array can receiveinputs of an arbitrary inputs and a reducer can reduce to a bit-lengthcorresponding to the maximum bit-length supported by elements of thesystolic array (e.g., a multiplier of a processing element). The binarymultiplier 534 may further perform floating-point multiplication,integer multiplication, or multiplication involving any other data type.The binary multiplier 534 may be implemented using a 16-bit multiplierdata path, an 18-bit multiplier data path, or a multiplier data pathwith any number of bits. The binary multiplier 534 may provide amultiplier product to the format adjuster 536. In some embodiments, thebinary multiplier 534 may be implemented using a multiplier circuit.

The format adjuster 536 may adjust the format of the multiplier productproduced by the binary multiplier 534. The significand data path of themultiplier 208 may include the format adjuster 536 to adjust the outputof the multiplier 208 into a format expected by one or more componentsof the systolic array (e.g., an adder separate from the adder 526). Forexample, an adder of the systolic array may expect (e.g., operate on) aninput with a 23-bit significand. The format adjuster 536 may add orreduce the number of bits used to represent the multiplier product, forexample, by increasing the bit size to 23 bits. The format adjuster 536may provide a partial significand data path 533 to the adder 210.

The adder 210 may contain a sign data path, an exponent data path, and asignificand data path. The adder 210 may be implemented with givenbit-size (e.g., with an adder data path of a given size). In someembodiments, each processing element may include an adder with a largerbit-size and a multiplier with a smaller bit-size as adders of increasedbit-sizes may be more cost efficient than multipliers of the sameincreased bit-sizes. Therefore, this disclose enables a systolic arrayto support, at reduced precision, larger bit-sizes using lower bit-sizemultipliers. The adder 210 may receive the partial sign data path 513,the partial exponent data path 523, and the partial significand datapath 533 from the multiplier 208. The adder 210 may also receive aninput partial sum 234. The adder 210 may perform an addition operationon the multiplier product comprised of the partial sign data path 513,the partial exponent data path 523, and the partial significand datapath 533 and the input partial sum 234. In some embodiments, the adder210 may perform addition operations on both floating-point and brainfloating-point numbers. Further, the adder 210 may be a 34-bitfloating-point adder, a 32-bit floating-point adder, or any otherbit-length adder.

The adder 210 may generate an addition result 238 based on the additionoperation. The addition result 238 may consist of a sign data path 515,an exponent data path 525, and a significand data path 535. In someembodiments, the addition result 238 may be an integer sum, afloating-point sum, or any other sum. Further, the adder 210 maygenerate a sum of 8-bits, 16-bits, 32-bits, 34-bits, or any other numberof bits. In some embodiments, the adder 210 may be implemented using abinary adder circuit.

FIG. 6 shows an apparatus 600 for neural network computations accordingto some embodiments of the disclosed technologies. The apparatus 600 maybe part of a computer system, e.g., a host server. For example, the hostserver may provide multi-tenant compute services for data processingapplications such as an image recognition service, text-based dataprocessing (e.g., processing of search queries), audio data processing,video data processing, etc. In some embodiments, a host device mayoperate a software application and communicate with the apparatus 600 tomake a prediction based on computations with a prediction modelutilizing a neural network processor. For example, the host device canmake the prediction by identifying information included in an input dataset for an image, text, audio, video, etc. using the prediction model.

The apparatus 600 may include a neural network processor 602 coupled tomemory 614, a host interface 616, and a direct memory access (DMA)controller 618 via an interconnect 620. The neural network processor 602may include a computing engine 604, a computation controller 606, astate buffer 608, an output buffer 610, and an activation engine 612.The neural network processor 602 can provide the computing resources tosupport the computations with the prediction model. The neural networkprocessor 602 may be implemented as a system on chip (SoC), a fieldprogrammable gate array (FPGA), or any suitable circuit.

The memory 614 may store instructions, input data sets (e.g., pixel dataof an image) and the weights (e.g., weights corresponding to certainvisual and/or non-visual features) received from the host device. Thememory 614 may also store outputs of the neural network processor 602(e.g., one or more image recognition decisions on the input images inthe form of output data sets). The memory 614 may include any suitablememory, e.g., dynamic random access memory (DRAM), synchronous DRAM(SDRAM), double data rate DRAM (DDR DRAM), storage class memory (SCM),flash memory, etc.

The host interface 616 may enable communication between the host deviceand the neural network processor 602. For example, the host interface616 may transmit memory descriptors including the memory addresses ofthe stored data (e.g., input data sets, weights, results ofcomputations, etc.) between the host device and the neural networkprocessor 602. The host interface 616 may include, e.g., a peripheralcomponent interconnect express (PCIe) interface, or any suitableinterface for communicating with the host device. The host device mayinclude a host processor and a host memory.

The DMA controller 618 may perform DMA operations to transfer databetween the neural network processor 602 and the host device. Forexample, as discussed above, the host device can store the instructions,input data sets, and the weights in the memory 614. The host device canprovide the memory addresses for the stored instructions, data, and theweights to the neural network processor 602 (e.g., in the form of memorydescriptors). The neural network processor 602 can then obtain thestored instructions, data, and the weights based on the memory addressesprovided by the host device. The neural network processor 602 can alsostore the results of computations (e.g., one or more image recognitiondecisions) in the memory 614, and provide the memory addresses for thestored results to the host device.

The state buffer 608 may provide caching of data used for computationsat the computing engine 604. The data cached at the state buffer 608 mayinclude, e.g., the input data sets and the weights acquired from thememory 614, as well as intermediate outputs of computations at thecomputing engine 604. The caching can reduce the effect of memory accessbottleneck (e.g., caused by the latencies at the memory 614, the DMAcontroller 618, the interconnect 620, etc.) on the performance of thecomputing engine 604. The state buffer 608 can be an on-chip memorydevice and may include a static random access memory (SRAM) or anysuitable memory.

The computation controller 606 may provide controls to variouscomponents of the neural network processor 602 to perform neural networkcomputations. In some implementations, the computation controller 606may read the instructions stored in the memory 614 and schedule theexecutions of the instructions by the computing engine 604. In the firstembodiment, the computation controller 606 may perform scheduling ofloading the weights into the computing engine 604 prior to reading theinput data elements from the state buffer 608. For example, as discussedwith reference to FIG. 2A, FIG. 2B, FIG. 4A, and FIG. 4B, thecomputation controller 606 may provide the opcode 230 and the weightload 232 to the computing engine 604 based on the instructions receivedfrom the host device. The computation controller 606 may provideappropriate values of the opcode 230 to the computing engine 604 whichmay be decoded by each PE in the computing engine 604 to perform acorresponding operation. For example, the computing engine 604 may usethe weight load 232 and the opcode 230 to pre-load the weights in allthe PEs in the computing engine 604. Once the weights have beenpre-loaded, the computation controller 606 may perform scheduling ofloading the input data elements into the computing engine 604,sequentially, in uniform time periods, from the state buffer 608 tostart the arithmetic computations.

In the second embodiment, the computation controller 606 may performscheduling of loading the weights and the input data elements into thecomputing engine 604, sequentially, in uniform time periods, from thestate buffer 608. The computation controller 606 may schedule loading ofthe weights and the input data elements in a respective first PE of eachrow in the systolic array 302 using a respective row data bus. Forexample, a respective input data element and a weight value may beloaded per cycle in the first PE of the respective row.

In another embodiment, the computation controller 606 may scheduleloading of the weights in the systolic array 302 in parallel for eachrow using a respective column data bus for each PE in a given row. Forexample, weights for each row may be loaded in parallel per cycle. Insome embodiments, the computation controller 606 may determine a datatype for the input data set based on the instructions received from thehost device. The instructions may be in the form of an opcode. The datatype may indicate a size and a type of the input data element, e.g.,4-bit, 8-bit, 16-bit, signed, unsigned, or floating-point.

The computing engine 604 may perform computations for the neuralnetwork. For example, the computing engine 604 may reduce the inputprovided to a systolic array to generate the reduced input. Further, thecomputing engine 604 may determine the maximum supported bit-length forthe systolic array and generate the reduced input with the maximumsupported bit-length. In some embodiments, the computing engine 604 mayinclude a set of PEs performing one or more arithmetic operationsinvolved in the neural network computations. Each PE may performmultiply-accumulate operations using input data sets and associatedweights. For example, the computing engine 604 may include the systolicarray 302, and the circuit 304 comprising the zero input data detectors306 a-306 x, and the zero weight detectors 308 a-308 x. In someembodiments, the zero input data detectors 306 a-306 x, and the zeroweight detectors 308 a-308 x may be external to the computing engine604. The computing engine 604 may execute instructions as scheduled bythe computation controller 606 to load the weights and the inputdatasets sequentially from the state buffer 608 into the computingengine 604.

In the first embodiment, the weights may be pre-loaded prior to readingthe input datasets from the state buffer 608, as discussed withreference to FIG. 4 . The respective zero weight indicatorscorresponding to each weight may be cached locally in each PE and thecached values may be used to perform arithmetic computations with therespective input data element as the input data element is fed into thecomputing engine 604 along with the corresponding zero data elementindicator. In the second embodiment, the weights and the input datasetsmay be read simultaneously from the state buffer 608, as discussed withreference to FIG. 5 . The corresponding zero data element indicator andthe zero weight indicator may be provided by the respective zerodetector circuits and propagated sequentially from one PE to another forthe respective row. The weights and the input datasets can be obtainedfrom the state buffer 608 using one or more interfaces. In certainembodiments, the computing engine 604 may perform the arithmeticcomputations to reduce the dynamic power consumption of the systolicarray 302 using the respective zero data element indicator and the zeroweight indicator signals as discussed with reference to FIGS. 2-5 , andprovide the computations results to be stored in the output buffer 610.

The output buffer 610 may include a set of registers to store the outputdata sets generated by the computing engine 604. In some embodiments,the output buffer 610 may also enable additional processing such as,e.g., a pooling operation to reduce the size of the stored outputs.Further, the computing engine 604 can be operated to performcomputations for a particular neural network layer, and the outputbuffer 610 can process the outputs of that neural network layer andstore the processed output datasets (with or without processing by theactivation engine 612) at the state buffer 608. The processed outputdatasets may be used by the computing engine 604 as the intermediateoutputs. In some embodiments, the output buffer 610 may include addersto accumulate the partial sums generated for different sets of filtersand input data sets to generate a convolution output array. The finaloutput value of the convolution output array stored in the state buffer608 can be retrieved by the computation controller 606 for storing atthe state buffer 608.

The activation engine 612 may apply one or more activation functions(e.g., ReLu function) on the output of the output buffer 610. Forexample, the activation engine 612 may include one or more lookup tables(e.g., in the form of multiplexer circuits) that can map the input toone of the candidate outputs representing the result of applying theactivation function to the input. In some examples, the activationengine 612 may also include a bypass path to allow outputs from theoutput buffer 610 to be stored directly at the state buffer 608 whenactivation functions are not to be applied.

FIG. 7 shows a method 700 executed by a computing engine 604 utilizing asystolic array (e.g., a group of processing elements), according to someexamples of the disclosed technologies. The array may be similar, forexample, to the array 100A, and include multiple PEs similar to, e.g.,the PE 112 a. The systolic array may include a plurality of PEsconfigured in a plurality of rows and/or a plurality of columns. Forexample, the systolic array might include 65,536 PEs which are furtherdivided into 256 rows and 256 columns. The computing engine 604 may be asystolic circuit that includes the systolic array and one or morereducers (e.g., convertors) to receive an input with an arbitrarybit-length and convert the arbitrary bit-length input into an input witha reduced bit-length corresponding to the maximum supported bit-lengthfor elements of the systolic array. For example, the one or morereducers can convert a plurality of input data elements (e.g., 32-bitinput data elements) into a plurality of reduced input data elements(e.g., 22-bit input data elements) and/or plurality of weights (e.g.,32-bit weights) into a plurality of reduced weights (e.g., 22-bitweights).

In block 702, a first reducer receives a first input (e.g., a firstnumber) with a first bit-length (e.g., 32 bits). The first inputbit-length may be an arbitrary bit-length. The first input may berepresented in floating-point format. Further, the first reducer canidentify a quantity of trailing bits of the first input and reduce thequantity of trailing bits of the first input. The first input mayrepresent an input data element. The first reducer may convert 32-bitfloating-point numbers to 22-bit floating-point numbers. In someembodiments, the first reducer may convert m-bit floating-point numbersto n-bit floating-point numbers, where n and m can be any numbers wheren is less than m.

In block 704, the first reducer generates a first reduced input with asecond bit-length (e.g., 22 bits). The second bit-length may be amaximum bit-length supported by elements of the systolic array. Forexample, the first reduced input may be a 22-bit floating-point number.Further, the second bit-length may be less than the first bit-length(e.g., the second bit-length may be any bit-length less than the firstbit-length). The first reducer may generate the first reduced inputbased on reducing the quantity of trailing bits of the first input. Togenerate the first reduced input (or any other reduced inputs), thefirst reducer may include a trailing bit reducer to reduce a quantity oftrailing bits representing a significand portion of the first input andproduce a reduced significand portion of the first input (e.g., the32-bit first input). For example, the trailing bit reducer may zero thequantity of trailing bits. Further, the first reducer may include arounder to round the reduced significand portion of the first inputbased at least in part on a remainder of the bits (e.g., a remainder ofnon-trailing bits of the first input) representing the significandportion of the first input not included within the reduced significandportion. For example, rounding the first input may include rounding aportion of the bits of the first input. The rounder may further roundthe first input to a particular number (e.g., a particularfloating-point number). In some embodiments, the rounder may round thesignificand portion and the trailing bit reducer may generate thereduced significand portion from the rounded significand portion (e.g.,the first input may be a first rounded input to the trailing bitreducer). In other embodiments, the first reducer may not include arounder and the significand portion may be pre-rounded (e.g., rounded byanother system) or not rounded). The rounder may round the input basedon one or more of stochastic rounding, rounding to nearest even,rounding to zero, rounding down, rounding up, or any other roundingmethod. Stochastic rounding may include rounding the input up to a firstnumber or down to a second number based on probabilities that are tunedbased on the relative distance between the input and the first numberand the relative distance between the input and the second numberrespectively. In some embodiments, the input may be rounded based onuser input (e.g., a selection of a rounding method). The first reducermay further include an exponent expander to increase a quantity of bitsrepresenting an exponent portion of the first input. In someembodiments, the first reduced input may be stored in a 24-bit format.

In some embodiments, the first reducer may generate a second input. Inother embodiments, the computing engine 604 may include a second reducerto receive a weight in floating-point format with the first bit-length.The second reducer may identify a quantity of trailing bits of theweight and reduce the quantity of trailing bits of the weight. Further,the second reducer may generate the weight in floating-point format withthe second bit-length based on reducing the quantity of trailing bits ofthe weight. For example, the second input may be a second 22-bitfloating-point number.

In block 706, an individual processing element in at least one row ofthe systolic array multiplies the first reduced input by the secondinput (e.g., a second number) to generate a multiplier product. In someembodiments, the second input may be a second reduced input. Forexample, the second input may be a reduced weight. The first reducer mayreceive the first input and a weight and generate the first reducedinput and the second input. Further, the first reducer can select thefirst reduced input or the second input to be provided to the individualprocessing element. The individual processing element may include amultiplier to multiply the first reduced input by the second input. Forexample, each processing element may include a 22-bit multiplier.Further, each processing element may include a multiplier to multiply atleast two inputs with the second bit-length (e.g., n-bit numbers).Further, the multiplier may multiply two 22-bit floating-point numbers.The multiplier may include a 1-bit sign data path, an 11-bit significanddata path, and a 10-bit exponent data path.

In block 708, the individual processing element adds an input partialsum with the multiplier product to generate an adder partial sum (e.g.,an addition result). The individual processing element may furtherinclude an adder to add the input partial sum with the multiplierproduct. For example, each processing element may include a 34-bitadder. Further, each processing element may include an adder to add atleast two numbers with a third bit-length (e.g., p-bit numbers where pis greater than n, the multiplier receiving n-bit numbers). Further, theadder may add two floating-point numbers. The adder may include a 1-bitsign data path, a 23-bit significand data path, and a 10-bit exponentdata path.

FIG. 8 shows a method 800 executed by a computing engine 604 utilizing asystolic array, according to some examples of the disclosedtechnologies. The array may be similar, for example, to the array 100A,and include multiple PEs similar to, e.g., the PE 112 a. The systolicarray may include a plurality of PEs configured in a plurality of rowsand/or a plurality of columns. For example, the systolic array mightinclude 65,536 PEs which are further divided into 256 rows and 256columns. The computing engine 604 may be a systolic circuit thatincludes the systolic array and one or more reducers (e.g., convertors)to receive an input with an arbitrary bit-length and convert thearbitrary bit-length input into multiple reduced inputs with a reducedbit-length corresponding to the maximum supported bit-length forelements of the systolic array. For example, the one or more reducerscan convert each of a plurality of input data elements (e.g., 32-bitinput data elements) into a multiple reduced input data elements (e.g.,21-bit input data elements) and/or each of a plurality of weights (e.g.,32-bit weights) into multiple reduced weights (e.g., 21-bit weights).

In block 802, the systolic array (e.g., a reducer of the systolic array)receives a first input (e.g., an input data element, a weight, etc.) infloating-point format with a first bit-length. For example, the firstinput may be a 32-bit floating-pint number. The systolic array may alsoreceive a second input (e.g., an input data element, a weight, etc.) formultiply-accumulate operations. The reducer may convert m-bitfloating-point numbers to one or more n-bit floating-point numbers,where n can be any number less than m. For example, the reducer canconvert 32-bit floating-point numbers to two 21-bit floating-pointnumbers.

In block 804, the systolic array generates a first reduced input (e.g.,a high reduced input) with a second bit-length. The first reduced inputmay correspond to a set of most significant bits of a significandportion of the first input (e.g., the leading bits of the significandportion of the first input).

In block 806, the systolic array generates a second reduced input (e.g.,a low reduced input) with a third bit-length. The second reduced inputmay correspond to a set of least significant bits of the significandportion of the first input (e.g., the trailing bits of the significandportion of the first input). The first reduced input and the secondreduced input may sum to the first input. Further, the second bit-lengthand the third bit-length may be less than the first bit-length from thefirst input. For example, the first reduced input and the second reducedinput may each be 21-bit floating-point numbers. Further, the reducermay convert an input data element and a weight into respective first andsecond reduced numbers.

Each of the first reduced input and the second reduced input may berepresented in floating-point format. In some embodiments, the reducermay generate the first reduced input and subtract the first reducedinput from the first input to generate the second reduced input. Forexample if the first input includes a first significand“11111111011010101010101,” the first reduced input includes a firstsignificand “11111111011,” by subtracting the first reduced input fromthe first input, the second reduced input may be determined as“010101010101.” The first reduced input and the second reduced input maybe a maximum supported bit-length for the systolic array and/or aparticular processing element. In some embodiments, the reducer mayinclude a first sub-reducer to generate the first reduced input. Thefirst sub-reducer may include a trailing bit reducer to reduce aquantity of trailing bits of a significand portion of the first input toproduce a high reduced significand portion. The first sub-reducer mayfurther include a first exponent expander to increase a quantity of bitsrepresenting an exponent portion of the first input to produce a firstincreased exponent portion. Based on the first increased exponentportion and the high reduced significand portion, the first sub-reducermay generate the first reduced input (e.g., the high reduced input).Further, the reducer may include a second sub-reducer to generate thesecond reduced input. The second sub-reducer may include a leading bitreducer to reduce a quantity of leading bits of a significand portion ofthe first input to produce a low reduced significand portion. The secondsub-reducer may further include a second exponent expander to increase aquantity of bits representing an exponent portion of the first input toproduce a second increased exponent portion. Based on the secondincreased exponent portion and the low reduced significand portion, thesecond sub-reducer may generate the second reduced input (e.g., the lowreduced input). In some embodiments, the second sub-reducer may alsoinclude a format detector to detect if the first input is denormal ornormal, a normalizer to remove an implied bit of the first input andrenormalize the low reduced significand portion to produce a normalizedsignificand portion, based on determining the first input is normal, andan exponent adjuster to adjust the second increased exponent portion toproduce an adjusted exponent portion based on renormalizing thesignificand portion. Further, the second reduced input may include theadjusted exponent portion and the normalized significand portion.

In block 808, the systolic array performs a plurality ofmultiply-accumulate operations on the first reduced input, the secondreduced input, and a second input. The first input may be an input dataelement or a weight and the second input may be the other of the inputdata element or the weight. In some embodiments, the second input maynot be reduced. In other embodiments, the systolic array may reduce thesecond input to generate a third reduced input and a fourth reducedinput for the plurality of multiply-accumulate operations. To performthe plurality of multiply-accumulate operations, the systolic array maycalculate a plurality of partial sums. Further, for each combination ofhigh/low reduced inputs, the systolic array can calculate a partial sum.For example, the systolic array can include processing elements toconduct multiply-accumulate operations on the reduced inputs. Theprocessing elements may each include a multiplier to multiply two 21-bitfloating-point numbers and an adder to add two floating-point numbers.Further, the multiplier may include a 1-bit sign data path, an 11-bitsignificand data path, and a 9-bit exponent data path and the adder mayinclude a 1-bit sign data path, a 23-bit significand data path, and a10-bit exponent data path. Further, the reducer may produce the reducedinputs and select the reduced inputs to be provided for processing bythe processing element. The plurality of operations may be a pluralityof ordered multiply-accumulate operations (e.g., a plurality of multiplyoperations and a plurality of accumulate operations for the firstinput). The processing element may include a multiplier to multiply atleast two n-bit number and an adder to add two p-bit numbers, where pmay be any number greater than n. For example, the multiplier be a21-bit multiplier to multiply two 21-bit numbers and the adder may be a34-bit adder. Further, to perform the operations, the processing elementcan multiply the second reduced input and a second reduced weight togenerate a first product, multiply the first reduced input and thesecond reduced weight to generate a second product, multiply the secondreduced input and the first reduced weight to generate a third product,multiply the second reduced input and the first reduced weight togenerate a fourth product, add the first product to an input partial sumto generate a first sum, add the first sum to the second product togenerate a second sum, add the second sum and the third product togenerate a third sum, and add the third sum and the fourth product togenerate a total product or output.

The systolic array may generate a full precision total output from theplurality of partial sums for the first input and the second input(e.g., the input data element and the weight) based on the reducedinputs. In some embodiments, to generate the total output, the systolicarray may provide each sub-product to an adder (e.g., an accumulator).The adder can perform chunk-based accumulation on the output of thesystolic array (e.g., each of the sub-products).

To better illustrate operation of a systolic array utilizing multiplecombinations of reduced inputs, FIG. 9A-9H illustrates an example fourPE column 900 of a systolic array for neural network computationsprocessing multiply-accumulate operations over systolic intervals 0through 9 according to certain examples of the disclosed technologies.The PE column 900 may be part of a systolic array similar to thesystolic array 100A in FIG. 1A, which may extend for any plurality ofrows and plurality of columns. In some embodiments, the systolic arraymay include a full multiply-accumulate operation for each combination ofreduced inputs (e.g., low input/weight and high input/weight) and theoutput of each operation may be summed.

The PE column 900 includes four PEs labeled as PE00, PE10, PE20, andPE30 according to their row and column (RC) number. In the example ofFIGS. 9A-9J, the column 900 is implementing two-pass multiply-accumulateoperations. For example, an input data element may be converted into tworeduced input data elements for multiply-accumulate operations. Theweight may be preloaded into the array and the weight may be used inmultiply-accumulate operations for each reduced input to generate anoutput. In some embodiments, the weight may also be converted into two(or any number of) reduced weights). A first reduced weight (e.g., thelow reduced weight) from the weight may be preloaded formultiply-accumulate operations with reduced input data elements and,subsequently, a second reduced weight (e.g., the high reduced weight)from the weight may be loaded for multiply-accumulate operations withthe same reduced input data elements. The output for each combination ofa reduced input and a reduced weight may be summed to generate a totaloutput. It will be understood that the column 900 may implement n-passmultiply accumulate operations where n can be any number. For example,the weight can be converted into any number of reduced weights and eachweight may iteratively loaded into the systolic array formultiply-accumulate operations with a set of reduced input dataelements.

Each PE illustratively includes a multiplier with a single systolicinterval latency (e.g., inputs provided at interval n are provided asoutputs at interval n+1) and an adder with a two-interval latency (e.g.,inputs provided at interval n are provided as outputs at interval n+2).Adders with other latencies may be implemented. As shown in FIGS. 9A-9H,each PE of the PE column 900 respectively includes a data register DataRegRC for receiving an input data element, a weight storing registerWeight RegRC, a multiplier represented by an “X”, and an adder oraccumulator represented by a “+”.

Values provided as input partial sums at systolic intervals 0-9 areshown along the top, with PE00 receiving values A1. (While value A1 isshown for illustrative purposes, in some instances all partial inputsums fed to a top row of an array may be set to the same value, whichmay be zero). Values provided as input data elements at systolicintervals 0-9 are shown along the left column, with PE00 in row 0receiving values C1 and C2 at the illustrated times, PE10 in row 1receiving values D1 and D2 at the illustrated times, PE20 in row 2receiving values E1 and E2 at the illustrated times, and PE30 in row 3receiving values F1 and F2 at the illustrated times. C1, D1, E1, and F1may each be a first reduced input data element (e.g., a low reducedinput data element) and C2, D2, E2, and F2 may each be a second reducedinput data element (e.g., a high reduced input data element). G1, H1,I1, and J1 may be the weight. In some embodiments, the weights may beeach converted into a first reduced weight (e.g., a low reduced weight)and a second reduced weight (e.g., a high reduced weight). When no valueis illustrated, a zero or NOP can be assumed. Where indicated, thesystem is initialized with zero values for clarity and to facilitateunderstanding. However, other examples can occur at different statesand/or with other internal values.

FIG. 9A-9H show the progression of data as multiply-accumulateoperations are performed. The multiply-accumulate operations across theshown intervals include (as discussed in more detail below): multiplyingweight G1 by input data element C1 and accumulating input partial sumA1; multiplying weight G1 by input data element C2; multiplying weightH1 by input data element D1 and accumulating input partial sum X1 fromPE00; multiplying weight H1 by input data element D2 and accumulatinginput partial sum X2 from PE00; multiplying weight I1 by input dataelement E1 and accumulating input partial sum Y1 from PE10; multiplyingweight I1 by input data element E2 and accumulating input partial sum Y2from PE10; multiplying weight J1 by input data element F1 andaccumulating input partial sum Z1 from PE20; and multiplying weight J1by input data element F2 and accumulating input partial sum Z2 fromPE20. The technology disclosed herein can extend to additional sequencesof input data elements and input partial sums.

FIG. 9A shows the state of the PE column 900 at systolic interval 0. Theweights G1, H1, I1, and J1 are each pre-loaded into respective weightregisters. For example, the weights G1, H1, I1, and J1 may be pre-loadedin a weight load operation. In PE00, an input data element C1 isreceived for writing to and storing in Data Reg00 for use during thenext systolic interval. All other inputs and other states areinitialized to zero.

FIG. 9B shows the state of the PE column 900 at systolic interval 1. InPE00, an input data element C2 is received for writing to and storing inData Reg00 for use during the next systolic interval. In someembodiments, the weight G1 may be preloaded into Weight Reg00 formultiply systolic intervals and may not be preloaded again. For example,the weight G1 may be preloaded for a plurality of multiply-accumulateoperations with a plurality of reduced input data elements. The weightG1 may subsequently be replaced with a new weight, G2, formultiply-accumulate operations with the reduced inputs. For example, G1and G2 may be reduced weights generated from a weight. Therefore, theweight G1 may only be preloaded into the array once. It will beunderstood that the combination of inputs or weights may be ordered suchthat any of the reduced inputs or weights may be stored in respectivedata registers for multiple systolic intervals and may not be rereadinto the PE. For example, the combinations of reduced inputs or weightsmay be ordered or distributed such that the weight G1 is not reread intothe PE. The stored input data element C1 is read from Data Reg00 andprovided as an input to both the multiplier of PE00 and a data registerof a PE in a subsequent column. The multiplier in PE00 multiplies C1 byG1 to generate a multiplication result C1×G1, which is provided to anadder for PE00. The input partial sum A1 is also received at the adderfor PE00. Each adder is pipelined with a latency of 2 intervals, and assuch processes the respective input partial sum and the respectivemultiplication result during a time period corresponding to the latency(e.g., the subsequent 2 intervals).

In PE10, an input data element D1 is received for writing to and storingin Data Reg10 for use during the next systolic interval.

FIG. 9C shows the state of the PE column 900 at systolic interval 2. InPE00, the input data element C2 is read from Data Reg00 and provided asan input to both the multiplier of PE00 and a data register of a PE in asubsequent column. The multiplier in PE00 multiplies C2 by G1 togenerate a multiplication result C2×G1, which is provided to the adderfor PE00 for use in an adder operation. Note that during systolicinterval 2, the adder of PE00 continues to conduct an add operationbetween the multiplication result C1×G1 and the input partial sum A1, asobtained during interval 1.

In PE10, an input data element D2 is received for writing to and storingin Data Reg10 for use during the next systolic interval. The storedinput data element D1 is read from Data Reg10 and provided as an inputto both the multiplier of PE10 and a data register of a PE in asubsequent column. The multiplier in PE10 multiplies D1 by H1 togenerate a multiplication result D1×H1, which is provided to an adderfor PE10.

In PE20, an input data element E1 is received for writing to and storingin Data Reg20 for use during the next systolic interval.

FIG. 9D shows the state of the PE column 900 at systolic interval 3. InPE00, the adder completes the addition of A1 and C1×G1 and generates anaddition result, A1+C1×G1. The addition result, A1+C1×G1, iscommunicated to PE10 as an input partial sum. The additional result of aPE within a given column can generally be referred to herein as a“partial sum.” Note that during systolic interval 3, the adder of PE00continues to conduct an add operation between the multiplication resultC2×G1, as obtained during interval 2.

In PE10, the stored input data element D2 is read from Data Reg10 andprovided as an input to both the multiplier of PE10 and a data registerof a PE in a subsequent column. The multiplier in PE10 multiplies D2 byH1 to generate a multiplication result D2×H1, which is provided to anadder for PE10. The input partial sum, C1×G1+A1, is received from PE00and is also provided to the adder for PE10 for use in the adderoperation. Note that during systolic interval 3, the adder of PE10continues to conduct an add operation between the multiplication resultD1×H1 and the input partial sum from PE00 (A1+C1×G1).

In PE20, an input data element E2 is received for writing to and storingin Data Reg20 for use during the next systolic interval. The storedinput data element E1 is read from Data Reg20 and provided as an inputto both the multiplier of PE20 and a data register of a PE in asubsequent column. The multiplier in PE20 multiplies E1 by I1 togenerate a multiplication result E1×I1, which is provided to the adderfor PE20 for use in an adder operation.

In PE30, an input data element F1 is received for writing to and storingin Data Reg30 for use during the next systolic interval.

FIG. 9E shows the state of the PE column 900 at systolic interval 4. theadder completes the addition of 0 and C2×G1 and generates an additionresult, C2×G1. In some embodiments, the input partial sum A1 may beadded to each combination of the reduced inputs. For example, where eachinput is converted into two reduced inputs resulting in fourcombinations of reduced inputs for each weight and input data element(e.g., a four-pass multiply-accumulate operation for a pair of inputs),the input partial sum may be added to each combination of reducedinputs. In other embodiments, a portion of the input partial sum may beadded to each combination of reduced inputs. For example, the inputpartial sum may be divided across each combination of reduced inputs.The addition result, C2×G1, is communicated to PE10 as an input partialsum.

In PE10, the input partial sum, C2×G1, is received from PE00 and is alsoprovided to the adder for PE10 for use in the adder operation. Note thatduring systolic interval 4, the adder of PE10 continues to conduct anadd operation between the multiplication result D2×H1 and the inputpartial sum from PE00 (C2×G1).

Further, in PE10, the adder completes the addition of D1×H1+C1×G1+A1 andgenerates an addition result, X1. The addition result, X1, iscommunicated to PE20 as an input partial sum.

In PE20, the stored input data element E2 is read from Data Reg20 andprovided as an input to both the multiplier of PE20 and a data registerof a PE in a subsequent column. The multiplier in PE20 multiplies E2 byI1 to generate a multiplication result E2×I1, which is provided to theadder for PE20 for use in an adder operation. The input partial sum, X1,is received from PE10 and is also provided to the adder for PE20 for usein the adder operation. Note that during systolic interval 4, the adderof PE20 continues to conduct an add operation between the multiplicationresult E1×I1 and the input partial sum from PE10 (X1).

In PE30, an input data element F2 is received for writing to and storingin Data Reg30 for use during the next systolic interval. The storedinput data element F1 is read from Data Reg30 and provided as an inputto both the multiplier of PE30 and a data register of a PE in asubsequent column. The multiplier in PE30 multiplies F1 by J1 togenerate a multiplication result F1×J1, which is provided to the adderfor PE30 for use in an adder operation.

FIG. 9F shows the state of the PE column 900 at systolic interval 5. InPE10, the adder completes the addition of D2×H1+C2×G1 and generates anaddition result, X2. The addition result, X2, is communicated to PE20 asan input partial sum.

In PE20, the input partial sum, X2, is received from PE10 and is alsoprovided to the adder for PE20 for use in the adder operation. Note thatduring systolic interval 5, the adder of PE20 continues to conduct anadd operation between the multiplication result E2×I1 and the inputpartial sum from PE10 (X2).

Further, in PE20, the adder completes the addition of E1×I1+X1 andgenerates an addition result, Y1. The addition result, Y1, iscommunicated to PE30 as an input partial sum.

In PE30, the stored input data element F2 is read from Data Reg30 andprovided as an input to both the multiplier of PE30 and a data registerof a PE in a subsequent column. The multiplier in PE30 multiplies F2 byJ1 to generate a multiplication result F2×J1, which is provided to theadder for PE30 for use in an adder operation. Note that during systolicinterval 5, the adder of PE30 continues to conduct an add operationbetween the multiplication result F1×J1, as obtained during interval 4and the input partial sum from PE20 (Y1).

FIG. 9G shows the state of the PE column 900 at systolic interval 6. InPE20, the adder completes the addition of E2×I1+X2 and generates anaddition result, Y2. The addition result, Y2, is communicated to PE30 asan input partial sum.

In PE30, the adder of PE30 continues to conduct an add operation betweenthe multiplication result F2×J1, as obtained during interval 5 and theinput partial sum from PE20 (Y2).

Further, in PE30, the adder completes the addition of F1×J1+Y1 andgenerates an addition result, Z1. The addition result, Z1, may becommunicated to another PE and/or to an aggregator for aggregation withadditional combinations of the reduced inputs for a particular set ofinputs.

FIG. 9H shows the state of the PE column 900 at systolic interval 7. InPE30, the adder completes the addition of F2×J1+Y2 and generates anaddition result, Z2. The addition result, Z2, may be communicated toanother PE and/or to an aggregator for aggregation with additionalcombinations of the reduced inputs for a particular set of inputs.

The examples states of data flow illustrated in FIG. 9A-9H can beperformed for one or more starting input data elements and for anynumber of starting input partial sums.

FIG. 10 illustrates an example of a computing device 1000. Functionalityand/or several components of the computing device 1000 may be usedwithout limitation with other embodiments disclosed elsewhere in thisdisclosure, without limitations. A computing device 1000 may performcomputations to facilitate processing of a task. As an illustrativeexample, computing device 1000 can be part of a server in a multi-tenantcompute service system. Various hardware and software resources ofcomputing device 1000 (e.g., the hardware and software resourcesassociated with data processing) can be allocated to a client uponrequest.

In one example, the computing device 1000 may include processing logic1002, a bus interface module 1004, memory 1006, and a network interfacemodule 1008. These modules may be hardware modules, software modules, ora combination of hardware and software. In certain instances, modulesmay be interchangeably used with components or engines, withoutdeviating from the scope of the disclosure. The computing device 1000may include additional modules, which are not illustrated here for theease of illustration. In some embodiments, the computing device 1000 mayinclude fewer modules. For example, one or more of the modules may becombined into one module. One or more of the modules may be incommunication with each other over a communication channel 1010. Thecommunication channel 1010 may include one or more busses, meshes,matrices, fabrics, a combination of these communication channels, orsome other suitable communication channel.

The processing logic 1002 may include application specific integratedcircuits (ASICs), field programmable gate arrays (FPGAs),systems-on-chip (SoCs), and network processing units (NPUs), processorsconfigured to execute instructions or any other circuitry to performlogical arithmetic and floating-point operations. Examples of processorsthat may be included in the processing logic 1002 may include processorsdeveloped by ARM®, MIPS®, AMD®, Qualcomm®, and the like. In someembodiments, the processors may include multiple processing cores andeach processing core may execute instructions independently of the otherprocessing cores. Further, each processor or processing core mayimplement multiple processing threads executing instructions on the sameprocessor or processing core, while maintaining logical separationbetween the multiple processing threads. Such processing threadsexecuting on the processor or processing core may be exposed to softwareas separate logical processors or processing cores. In some embodiments,multiple processors, processing cores or processing threads executing onthe same core may share certain resources, such as for example busses,level 1 (L1) caches, and/or level 2 (L2) caches. The instructionsexecuted by the processing logic 1002 may be stored on acomputer-readable storage medium, for example, in the form of a computerprogram. The computer-readable storage medium may be non-transitory. Insome cases, the computer-readable medium may be part of the memory 1006.The processing logic 1002 may also include hardware circuities forperforming artificial neural network computations including, forexample, the neural network processor 602, etc.

The access to the processing logic 1002 can be granted to a client toprovide the personal assistant service requested by the client. Forexample, the computing device 1000 may host a virtual machine, on whichan image recognition software application can be executed. The imagerecognition software application, upon execution, may access theprocessing logic 1002 to predict, for example, an object included in animage. As another example, access to the processing logic 1002 can alsobe granted as part of bare-metal instance, in which an image recognitionsoftware application executing on a client device (e.g., a remotecomputer, a smart phone, etc.) can directly access the processing logic1002 to perform the recognition of an image.

The memory 1006 may include either volatile or non-volatile, or bothvolatile and non-volatile types of memory. The memory 1006 may, forexample, include random access memory (RAM), read only memory (ROM),Electrically Erasable Programmable Read-Only Memory (EEPROM), flashmemory, and/or some other suitable storage media. In some cases, some orall of the memory 1006 may be internal to the computing device 1000,while in other cases some or all of the memory may be external to thecomputing device 1000. The memory 1006 may store an operating systemcomprising executable instructions that, when executed by the processinglogic 1002, provides the execution environment for executinginstructions providing functionality to perform convolution computationsfor the computing device 1000. The memory 1006 may also store, forexample, software applications for performing artificial neural networkcomputations. The memory may also store and maintain several datastructures and tables for facilitating the functionality of thecomputing device 1000.

The bus interface module 1004 may enable communication with externalentities, such as a host device and/or other components in a computingsystem, over an external communication medium. The bus interface module1004 may include a physical interface for connecting to a cable, socket,port, or other connection to the external communication medium. The businterface module 1004 may further include hardware and/or software tomanage incoming and outgoing transactions. The bus interface module 1004may implement a local bus protocol, such as Peripheral ComponentInterconnect (PCI) based protocols, Non-Volatile Memory Express (NVMe),Advanced Host Controller Interface (AHCI), Small Computer SystemInterface (SCSI), Serial Attached SCSI (SAS), Serial AT Attachment(SATA), Parallel ATA (PATA), some other standard bus protocol, or aproprietary bus protocol. The bus interface module 1004 may include thephysical layer for any of these bus protocols, including a connector,power management, and error handling, among other things. In someembodiments, the computing device 1000 may include multiple businterface modules for communicating with multiple external entities.These multiple bus interface modules may implement the same local busprotocol, different local bus protocols, or a combination of the sameand different bus protocols.

The network interface module 1008 may include hardware and/or softwarefor communicating with a network. This network interface module 1008may, for example, include physical connectors or physical ports forwired connection to a network, and/or antennas for wirelesscommunication to a network. The network interface module 1008 mayfurther include hardware and/or software implementing a network protocolstack. The network interface module 1008 may communicate with thenetwork using a network protocol, such as for example TCP/IP,Infiniband, RoCE, Institute of Electrical and Electronics Engineers(IEEE) 802.11 wireless protocols, User Datagram Protocol (UDP),Asynchronous Transfer Mode (ATM), token ring, frame relay, High LevelData Link Control (HDLC), Fiber Distributed Data Interface (FDDI),and/or Point-to-Point Protocol (PPP), among others. In some embodiments,the computing device 1000 may include multiple network interfacemodules, each configured to communicate with a different network. Forexample, the computing device 1000 may include a network interfacemodule for communicating with a wired Ethernet network, a wireless1002.11 network, a cellular network, an Infiniband network, etc. In someembodiments, the computing device 1000 may receive a set of parameters,such as the aforementioned weight values for convolution computations,from a server through network interface module 1008.

The various components and modules of the computing device 1000,described above, may be implemented as discrete components, as a Systemon a Chip (SoC), as an ASIC, as an NPU, as an FPGA, or any combinationthereof. In some embodiments, the SoC or other component may becommunicatively coupled to another computing system to provide variousservices such as traffic monitoring, traffic shaping, computing, etc. Insome embodiments of the technology, the SoC or other component mayinclude multiple subsystems as disclosed herein.

The modules described herein may be software modules, hardware modulesor a suitable combination thereof. If the modules are software modules,the modules can be embodied on a non-transitory computer readable mediumand processed by a processor in any of the computer systems describedherein. It should be noted that the described processes andarchitectures can be performed either in real-time or in an asynchronousmode prior to any user interaction. The modules may be configured in themanner suggested in FIG. 10 , and/or functions described herein can beprovided by one or more modules that exist as separate modules and/ormodule functions described herein can be spread over multiple modules.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the disclosure asset forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit thedisclosure to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructions,and equivalents falling within the spirit and scope of the disclosure,as defined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected” is to be construed as partly or wholly contained within,attached to, or joined together, even if there is something intervening.Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate embodiments of the disclosure anddoes not pose a limitation on the scope of the disclosure unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe disclosure.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is intended to be understoodwithin the context as used in general to present that an item, term,etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y,and/or Z). Thus, such disjunctive language is not generally intended to,and should not, imply that certain embodiments require at least one ofX, at least one of Y, or at least one of Z to each be present.

Various embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the disclosure.Variations of those embodiments may become apparent to those of ordinaryskill in the art upon reading the foregoing description. The inventorsexpect skilled artisans to employ such variations as appropriate and theinventors intend for the disclosure to be practiced otherwise than asspecifically described herein. Accordingly, this disclosure includes allmodifications and equivalents of the subject matter recited in theclaims appended hereto as permitted by applicable law. Moreover, anycombination of the above-described elements in all possible variationsthereof is encompassed by the disclosure unless otherwise indicatedherein or otherwise clearly contradicted by context.

Various example embodiments of the disclosure can be described by thefollowing clauses:

-   -   Clause 1: A systolic array processor organized in rows and        columns, each row comprising:        -   a reducer, the reducer configured to convert 32-bit input            data elements into reduced 22-bit input data elements, the            reducer comprising:            -   a trailing bit reducer configured to reduce a quantity                of bits representing a significand portion of a 32-bit                input data element of the 32-bit input data elements to                produce a reduced significand portion of the 32-bit                input data element;            -   a rounder configured to round the reduced significand                portion of the 32-bit input data element to produce a                rounded significand portion; and            -   an exponent expander configured to increase a quantity                of bits representing an exponent portion of the 32-bit                input data element to produce an increased exponent                portion,            -   wherein the reducer produces a reduced 22-bit input data                element based on the rounded significand portion and the                increased exponent portion; and        -   a plurality of processing elements, the plurality of            processing elements configured to receive the reduced 22-bit            input data elements from the reducer and to receive weights            for performing multiply-accumulate operations.    -   Clause 2: The systolic array processor of Clause 1, wherein the        reducer is further configured to convert 32-bit weights into the        weights.    -   Clause 3: The systolic array processor of Clause 1, wherein the        reducer further comprises a first reducer, each row further        comprising:        -   a second reducer, the second reducer configured to convert            32-bit weights into the weights.    -   Clause 4: The systolic array processor of Clause 1, wherein the        rounder is configured to round the reduced significand portion        of the 32-bit input data element based on one or more of:        -   stochastic rounding;        -   rounding to nearest even;        -   rounding to zero;        -   rounding down; or        -   rounding up.    -   Clause 5: A systolic circuit comprising:        -   a group of processing elements arranged into a plurality of            rows; and        -   a first convertor configured to:            -   receive a first input represented in floating-point with                a first bit-length;            -   identify a quantity of trailing bits of the first input;            -   reducing the quantity of trailing bits of the first                input; and            -   generate a first reduced input represented in                floating-point with a second bit-length based on                reducing the quantity of trailing bits of the first                input, wherein the second bit-length is less than the                first bit-length, wherein the second bit-length                corresponds to a bit-length supported by the group of                processing elements;        -   wherein an individual processing element in at least one row            of the group of processing elements is configured to receive            the first reduced input from the first convertor and to            receive a second input for performing multiply-accumulate            operations.    -   Clause 6: The systolic circuit of Clause 5, wherein individual        processing elements in the plurality of rows of the group of        processing elements comprise:        -   a multiplier configured to multiply two 22-bit            floating-point numbers, wherein the multiplier is comprised            of a 1-bit sign data path, a 11-bit significand data path,            and a 10-bit exponent data path; and        -   an adder configured to add two floating-point numbers,            wherein the adder is comprised of a 1-bit sign data path, a            23-bit significand data path, and a 10-bit exponent data            path.    -   Clause 7: The systolic circuit of Clause 5, wherein the first        input comprises an input data element and the second input        comprises a reduced weight, wherein the first convertor is        further configured to:        -   receive the first input and a weight;        -   generate the first reduced input and the second input; and        -   select the first reduced input or the second input to be            provided.    -   Clause 8: The systolic circuit of Clause 5, wherein the first        convertor comprises:        -   a trailing bit reducer configured to reduce a quantity of            bits representing a significand portion of the first input            to produce a reduced significand portion of the first input;        -   a rounder configured to round the reduced significand            portion of the first input based on a remainder of the bits            representing the significand portion of the first input not            included within the reduced significand portion; and        -   an exponent expander configured to increase a quantity of            bits representing an exponent portion of the first input.    -   Clause 9: The systolic circuit of Clause 5, wherein the first        input comprises a first rounded input, wherein the first        convertor comprises:        -   a trailing bit reducer configured to reduce a quantity of            bits representing a significand portion of the first input            to produce a reduced significand portion of the first input;            and        -   an exponent expander configured to increase a quantity of            bits representing an exponent portion of the first input.    -   Clause 10: The systolic circuit of Clause 5, wherein the first        reduced input comprises a first reduced rounded input, wherein        the first reduced rounded input is rounded based on one or more        of:        -   stochastic rounding;        -   rounding to nearest even;        -   rounding to zero;        -   rounding down; or        -   rounding up.    -   Clause 11: The systolic circuit of Clause 5, wherein the first        reduced input comprises a first reduced rounded input, wherein        the first reduced rounded input is rounded based on a user        input.    -   Clause 12: The systolic circuit of Clause 5, wherein:        -   the first convertor is configured to convert 32-bit            floating-point numbers to 22-bit floating-point numbers,        -   wherein each of the processing elements comprises:            -   a 22-bit multiplier; and            -   a 34-bit adder.    -   Clause 13: The systolic circuit of Clause 5, wherein:        -   the first convertor is further configured to convert m-bit            floating-point numbers to n-bit floating-point numbers,            wherein n and m can be any positive integer, wherein n is            less than m,        -   wherein each of the processing elements comprises:            -   a multiplier configured to multiply at least two n-bit                numbers; and            -   an adder configured to add two p-bit numbers, wherein p                is greater than n.    -   Clause 14: The systolic circuit of Clause 5, wherein to reduce        the quantity of trailing bits of the first input, the first        convertor is configured to:        -   set the quantity of trailing bits to zero.    -   Clause 15: The systolic circuit of Clause 5, further comprising:        -   a second convertor configured to:            -   receive a weight represented in floating-point with the                first bit-length;            -   identify a quantity of trailing bits of the weight;            -   reduce the quantity of trailing bits of the weight; and            -   generate the second input represented in floating-point                with the second bit-length based on reducing the                quantity of trailing bits of the weight.    -   Clause 16: The systolic circuit of Clause 5, wherein the first        reduced input is stored in a 24-bit format.    -   Clause 17: A method, comprising:        -   receiving a first input represented in floating-point with a            first bit-length;        -   reducing a quantity of trailing bits of the first input;        -   generating a first reduced input represented in            floating-point with a second bit-length based on reducing            the quantity of trailing bits of the first input, wherein            the second bit-length is less than the first bit-length,            wherein the second bit-length corresponds to a supported            bit-length; and        -   receiving the first reduced input and a second input for            performing multiply-accumulate operations.    -   Clause 18: The method of Clause 17, wherein:        -   the first input comprises a 32-bit floating-point number;        -   the first reduced input comprises a first 22-bit            floating-point number; and        -   the second input comprises a second 22-bit floating-point            number.    -   Clause 19: The method of Clause 17, wherein generating the first        reduced input comprises:        -   rounding the first input to generate the first reduced            input, based on a remainder of non-trailing bits of the            first input, wherein the first input comprises a quantity of            bits, wherein rounding the first input comprises rounding a            portion of the quantity of bits.    -   Clause 20: The method of Clause 17, wherein one or more of the        first reduced input or the second input comprises a rounded,        reduced input, wherein the rounded, reduced input is rounded        based on one or more of:        -   stochastic rounding;        -   rounding to nearest even;        -   rounding to zero;        -   rounding down; or        -   rounding up.

Various example embodiments of the disclosure can be described by thefollowing clauses:

-   -   Clause 1: A systolic array processor organized in rows and        columns, each row comprising:        -   a reducer configured to convert 32-bit input data elements            into two 21-bit input data elements, the reducer comprising:            -   a first sub-reducer configured to convert a 32-bit input                data element of the 32-bit input data elements into a                first 21-bit input data element, the first 21-bit input                data element corresponding to a set of most significant                bits of a significand portion of the 32-bit input data                element, the first sub-reducer comprising:                -   a trailing bit reducer configured to reduce a                    quantity of trailing bits representing the                    significand portion of the 32-bit input data element                    to produce a first reduced significand portion of                    the 32-bit input data element, the first reduced                    significand portion corresponding to the set of most                    significant bits; and                -   a first exponent expander configured to increase a                    quantity of bits representing an exponent portion of                    the 32-bit input data element to produce a first                    increased exponent portion,                -   wherein the first sub-reducer produces the first                    21-bit input data element based on the first reduced                    significand portion and the first increased exponent                    portion; and            -   a second sub-reducer configured to convert the 32-bit                input data element into a second 21-bit input data                element, the second 21-bit input data element                corresponding to a set of least significant bits of the                significand portion of the 32-bit input data element,                the second sub-reducer comprising:                -   a leading bit reducer configured to reduce a                    quantity of leading bits representing the                    significand portion of the 32-bit input data element                    to produce a second reduced significand portion of                    the 32-bit input data element, the second reduced                    significand portion corresponding to the set of                    least significant bits; and                -   a second exponent expander configured to increase a                    quantity of bits representing the exponent portion                    of the 32-bit input data element to produce a second                    increased exponent portion,                -   wherein the second sub-reducer produces a second                    21-bit input data element based on the second                    reduced significand portion and the second increased                    exponent portion; and        -   a plurality of processing elements, a processing element of            the plurality of processing elements configured to            iteratively perform a plurality of pairwise            multiply-accumulate operations on the first 21-bit input            data element, the second 21-bit input data element, and a            weight to provide a total output, wherein a 21 bit-length            corresponds to a maximum supported bit-length for the            processing element.    -   Clause 2: The systolic array processor of Clause 1, wherein the        first 21-bit input data element and the second 21-bit input data        element sum to the 32-bit input data element.    -   Clause 3: The systolic array processor of Clause 1, wherein the        second sub-reducer is further configured to determine the 32-bit        input data element comprises a normal number, the second        sub-reducer further comprising:        -   a normalizer to remove an implied bit of the 32-bit input            data element and renormalize the second reduced significand            portion to produce a normalized significand portion based on            determining the 32-bit input data element comprises a normal            number; and        -   an exponent adjuster to adjust the second increased exponent            portion to produce an adjusted exponent portion based on            renormalizing the second reduced significand portion,        -   wherein the second 21-bit input data element is further            based on the normalized significand portion and the adjusted            exponent portion.    -   Clause 4: The systolic array processor of Clause 1, the weight        comprising a first reduced weight and a second reduced weight,        wherein the processing element is further configured to:        -   multiply the second 21-bit input data element and the second            reduced weight to generate a first product;        -   multiply the first 21-bit input data element and the second            reduced weight to generate a second product;        -   multiply the second 21-bit input data element and the first            reduced weight to generate a third product; and        -   multiply the first 21-bit input data element and the first            reduced weight to generate a fourth product,    -   wherein the systolic array processor further comprises a partial        sum buffer configured to:        -   add the first product, the second product, the third            product, the fourth product, and an input partial sum to            generate the total output.    -   Clause 5: A systolic circuit comprising:        -   a group of processing elements arranged into a plurality of            rows; and        -   a first convertor configured to:            -   receive a first input represented in floating-point with                a first bit-length;            -   generate a first reduced input represented in                floating-point with a second bit-length, the first                reduced input corresponding to a set of most significant                bits of a significand portion of the first input; and            -   generate a second reduced input represented in                floating-point with a third bit-length, the second                reduced input corresponding to a set of least                significant bits of the significand portion of the first                input, wherein the first reduced input and the second                reduced input sum to the first input, wherein the second                bit-length and the third bit-length are less than the                first bit-length, wherein the second bit-length and the                third bit-length correspond to a bit-length supported by                the group of processing elements,        -   wherein an individual processing element in at least one row            of the group of processing elements is configured to receive            the first reduced input and the second reduced input and            perform a plurality of multiply-accumulate operations on the            first reduced input, the second reduced input, and a second            input.    -   Clause 6: The systolic circuit of Clause 5, wherein individual        processing elements in the plurality of rows of the group of        processing elements comprise:        -   a multiplier configured to multiply two 21-bit            floating-point numbers, wherein the multiplier is comprised            of a 1-bit sign data path, a 11-bit significand data path,            and a 9-bit exponent data path; and        -   an adder configured to add two floating-point numbers,            wherein the adder is comprised of a 1-bit sign data path, a            23-bit significand data path, and a 10-bit exponent data            path.    -   Clause 7: The systolic circuit of Clause 5, wherein the first        input corresponds to an input data element and the second input        corresponds to a weight, wherein the first convertor is further        configured to:        -   receive the second input represented in floating-point with            a fourth bit-length;        -   generate a third reduced input represented in floating-point            with a fifth bit-length, the third reduced input            corresponding to a set of most significant bits of a            significand portion of the second input;        -   generate a fourth reduced input represented in            floating-point with a sixth bit-length, the fourth reduced            input corresponding to a set of least significant bits of            the significand portion of the second input, wherein the            third reduced input and the fourth reduced input sum to the            second input, wherein the fifth bit-length and the sixth            bit-length are less than the fourth bit-length, wherein the            fifth bit-length and the sixth bit-length correspond to the            bit-length supported by the group of processing elements;            and        -   select the first reduced input, the second reduced input,            the third reduced input, or the fourth reduced input to be            provided.    -   Clause 8: The systolic circuit of Clause 5, wherein the first        convertor comprises:        -   a first sub-reducer comprising:            -   a trailing bit reducer configured to reduce a quantity                of the set of least significant bits of the significand                portion of the first input to produce a first reduced                significand portion of the first input; and            -   a first exponent expander configured to increase a                quantity of bits representing an exponent portion of the                first input to produce a first increased exponent                portion,            -   wherein the first sub-reducer produces the first reduced                input based on the first reduced significand portion and                the first increased exponent portion; and        -   a second sub-reducer comprising:            -   a leading bit reducer configured to reduce a quantity of                the set of most significant bits of the significand                portion of the first input to produce a second reduced                significand portion of the first input; and            -   a second exponent expander configured to increase a                quantity of bits representing the exponent portion of                the first input to produce a second increased exponent                portion,            -   wherein the second sub-reducer produces the second                reduced input based on the second reduced significand                portion and the second increased exponent portion.    -   Clause 9: The systolic circuit of Clause 8, wherein the second        sub-reducer is configured to determine the first input comprises        a normal number, the second sub-reducer further comprising:        -   a normalizer to remove an implied bit of the first input and            renormalize the second reduced significand portion to            produce a normalized significand portion based on            determining the first input comprises a normal number; and        -   an exponent adjuster to adjust the second increased exponent            portion to produce an adjusted exponent portion based on            renormalizing the second reduced significand portion,        -   wherein the second reduced input is further based on the            normalized significand portion and the adjusted exponent            portion.    -   Clause 10: The systolic circuit of Clause 5, wherein the second        input corresponds to a first reduced weight and a second reduced        weight, wherein to perform the plurality of multiply-accumulate        operations, the individual processing element is configured to:        -   multiply the second reduced input and the second reduced            weight to generate a first product;        -   add the first product to an input partial sum to generate a            first sum;        -   multiply the first reduced input and the second reduced            weight to generate a second product;        -   multiply the second reduced input and the first reduced            weight to generate a third product; and        -   multiply the first reduced input and the first reduced            weight to generate a fourth product,    -   wherein the systolic circuit further comprises a partial sum        buffer configured to:        -   add the first sum to the second product to generate a second            sum;        -   add the second sum and the third product to generate a third            sum; and        -   add the third sum and the fourth product to generate a total            output.    -   Clause 11: The systolic circuit of Clause 5, wherein the        plurality of multiply-accumulate operations comprise an ordered        plurality of multiply-accumulate operations.    -   Clause 12: The systolic circuit of Clause 5, wherein:        -   the first convertor is configured to convert 32-bit            floating-point numbers to a plurality of 22-bit            floating-point numbers,        -   wherein each of the processing elements comprises:            -   a 22-bit multiplier; and            -   a 34-bit adder.    -   Clause 13: The systolic circuit of Clause 5, wherein:        -   the first convertor is further configured to convert m-bit            floating-point numbers to one or more n-bit floating-point            numbers, wherein n and m can be any number, wherein n is            less than m,        -   wherein each of the processing elements comprises:            -   a multiplier configured to multiply at least two n-bit                numbers; and            -   an adder configured to add two p-bit numbers, wherein p                is greater than n.    -   Clause 14: The systolic circuit of Clause 5, the systolic        circuit further comprising:        -   a partial sum buffer configured to perform chunk-based            accumulation based on a plurality of outputs of the group of            processing elements.    -   Clause 15: The systolic circuit of Clause 5, further comprising:        -   a second convertor configured to:            -   receive the second input represented in floating-point                with a fourth bit-length, the second input corresponding                to a weight;            -   generate a third reduced input represented in                floating-point with a fifth bit-length, the third                reduced input corresponding to a set of most significant                bits of a significand portion of the second input; and            -   generate a fourth reduced input represented in                floating-point with a sixth bit-length, the fourth                reduced input corresponding to a set of least                significant bits of the significand portion of the                second input, wherein the third reduced input and the                fourth reduced input sum to the second input, wherein                the fifth bit-length and the sixth bit-length are less                than the fourth bit-length, wherein the fifth bit-length                and the sixth bit-length correspond to the bit-length                supported by the group of processing elements,        -   wherein the individual processing element in the at least            one row of the group of processing elements is further            configured to receive the third reduced input and the fourth            reduced input and perform the plurality of            multiply-accumulate operations on the first reduced input,            the second reduced input, the third reduced input, and the            fourth reduced input.    -   Clause 16: The systolic circuit of Clause 5, wherein the group        of processing elements perform a first accumulation on a        plurality of outputs of the group of processing elements to        produce a reduced plurality of outputs, the systolic circuit        further comprising:        -   a partial sum buffer configured to perform chunk-based            accumulation based on the reduced plurality of outputs to            generate an output.    -   Clause 17: A method, comprising:        -   receiving a first input represented in floating-point;        -   generating a first reduced input represented in            floating-point, the first reduced input corresponding to a            set of most significant bits of a significand portion of the            first input;        -   generating a second reduced input represented in            floating-point, the second reduced input corresponding to a            set of least significant bits of the significand portion of            the first input, wherein the first reduced input and the            second reduced input sum to the first input, wherein the            first reduced input and the second reduced input correspond            to a supported bit-length; and        -   performing one or more operations based on the first reduced            input, the second reduced input, and a second input to            generate an output.    -   Clause 18: The method of Clause 17, wherein:        -   the first input comprises a 32-bit floating-point number;        -   the first reduced input comprises a first 22-bit            floating-point number; and        -   the second reduced input comprises a second 22-bit            floating-point number.    -   Clause 19: The method of Clause 17, further comprising:        -   receiving the second input represented in floating-point;        -   generating a third reduced input represented in            floating-point, the third reduced input corresponding to a            set of most significant bits of a significand portion of the            second input; and        -   generating a fourth reduced input represented in            floating-point, the fourth reduced input corresponding to a            set of least significant bits of the significand portion of            the second input, wherein the third reduced input and the            fourth reduced input sum to the second input,        -   wherein the one or more operations are further based on the            third reduced input and the fourth reduced input.    -   Clause 20: The method of Clause 17, wherein each of the first        input and the second input comprises an input data element or a        weight.

The processes described herein or illustrated in the figures of thepresent disclosure may begin in response to an event, such as on apredetermined or dynamically determined schedule, on demand wheninitiated by a user or system administrator, or in response to someother event. When such processes are initiated, a set of executableprogram instructions stored on one or more non-transitorycomputer-readable media (e.g., hard drive, flash memory, removablemedia, etc.) may be loaded into memory (e.g., RAM) of a server or othercomputing device. The executable instructions may then be executed by ahardware-based computer processor of the computing device. In someembodiments, such processes or portions thereof may be implemented onmultiple computing devices and/or multiple processors, serially or inparallel.

Depending on the embodiment, certain acts, events, or functions of anyof the processes or algorithms described herein can be performed in adifferent sequence, can be added, merged, or left out altogether (e.g.,not all described operations or events are necessary for the practice ofthe algorithm). Moreover, in certain embodiments, operations or eventscan be performed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware (e.g., ASICs or FPGAdevices), computer software that runs on computer hardware, orcombinations of both. A processor device can be a microprocessor, but inthe alternative, the processor device can be a controller,microcontroller, or state machine, combinations of the same, or thelike. A processor device can include electrical circuitry to processcomputer-executable instructions. In another embodiment, a processordevice includes an FPGA or other programmable device that performs logicoperations without processing computer-executable instructions. Aprocessor device can also be implemented as a combination of computingdevices, e.g., a combination of a DSP and a microprocessor, a pluralityof microprocessors, one or more microprocessors in conjunction with aDSP core, or any other such configuration. Although described hereinprimarily with respect to digital technology, a processor device mayalso include primarily analog components. For example, some or all ofthe rendering techniques described herein may be implemented in analogcircuitry or mixed analog and digital circuitry. A computing environmentcan include any type of computer system, including, but not limited to,a computer system based on a microprocessor, a mainframe computer, adigital signal processor, a portable computing device, a devicecontroller, or a computational engine within an appliance, to name afew.

The elements of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processordevice, or in a combination of the two. A software module can reside inRAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, hard disk, a removable disk, a CD-ROM, or any other form of anon-transitory computer-readable storage medium. An exemplary storagemedium can be coupled to the processor device such that the processordevice can read information from, and write information to, the storagemedium. In the alternative, the storage medium can be integral to theprocessor device. The processor device and the storage medium can residein an ASIC. The ASIC can reside in a user terminal. In the alternative,the processor device and the storage medium can reside as discretecomponents in a user terminal.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements or steps.Thus, such conditional language is not generally intended to imply thatfeatures, elements or steps are in any way required for one or moreembodiments or that one or more embodiments necessarily include logicfor deciding, with or without other input or prompting, whether thesefeatures, elements or steps are included or are to be performed in anyparticular embodiment. The terms “comprising,” “including,” “having,”and the like are synonymous and are used inclusively, in an open-endedfashion, and do not exclude additional elements, features, acts,operations, and so forth. Also, the term “or” is used in its inclusivesense (and not in its exclusive sense) so that when used, for example,to connect a list of elements, the term “or” means one, some, or all ofthe elements in the list.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y, or Z, or any combination thereof (e.g., X, Y, or Z). Thus,such disjunctive language is not generally intended to, and should not,imply that certain embodiments require at least one of X, at least oneof Y, and at least one of Z to each be present.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it can beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the scope of the disclosure. As can berecognized, certain embodiments described herein can be embodied withina form that does not provide all of the features and benefits set forthherein, as some features can be used or practiced separately fromothers. All changes which come within the meaning and range ofequivalency of the Clauses are to be embraced within their scope.

What is claimed is:
 1. A systolic array processor organized in rows andcolumns, each row comprising: a reducer, the reducer configured toconvert 32-bit input data elements into reduced 22-bit input dataelements, the reducer comprising: a trailing bit reducer configured toreduce a quantity of bits representing a significand portion of a 32-bitinput data element of the 32-bit input data elements to produce areduced significand portion of the 32-bit input data element; a rounderconfigured to round the reduced significand portion of the 32-bit inputdata element to produce a rounded significand portion; and an exponentexpander configured to increase a quantity of bits representing anexponent portion of the 32-bit input data element to produce anincreased exponent portion, wherein the reducer produces a reduced22-bit input data element based on the rounded significand portion andthe increased exponent portion; and a plurality of processing elements,the plurality of processing elements configured to receive the reduced22-bit input data elements from the reducer and to receive weights forperforming multiply-accumulate operations.
 2. The systolic arrayprocessor of claim 1, wherein the reducer is further configured toconvert 32-bit weights into the weights.
 3. The systolic array processorof claim 1, wherein the reducer further comprises a first reducer, eachrow further comprising: a second reducer, the second reducer configuredto convert 32-bit weights into the weights.
 4. The systolic arrayprocessor of claim 1, wherein the rounder is configured to round thereduced significand portion of the 32-bit input data element based onone or more of: stochastic rounding; rounding to nearest even; roundingto zero; rounding down; or rounding up.
 5. A systolic circuitcomprising: a group of processing elements arranged into a plurality ofrows; and a first convertor configured to: receive a first inputrepresented in floating-point with a first bit-length; identify aquantity of trailing bits of the first input based on a differencebetween the first bit-length and a bit-length supported by the group ofprocessing elements; reduce the quantity of trailing bits of the firstinput; and generate a first reduced input represented in floating-pointwith a second bit-length based on reducing the quantity of trailing bitsof the first input, wherein the second bit-length is less than the firstbit-length, wherein the second bit-length corresponds to a bit-lengthsupported by the group of processing elements; wherein an individualprocessing element in at least one row of the group of processingelements is configured to receive the first reduced input from the firstconvertor and to receive a second input for performingmultiply-accumulate operations.
 6. The systolic circuit of claim 5,wherein individual processing elements in the plurality of rows of thegroup of processing elements comprise: a multiplier configured tomultiply two 22-bit floating-point numbers, wherein the multiplier iscomprised of a 1-bit sign data path, a 11-bit significand data path, anda 10-bit exponent data path; and an adder configured to add twofloating-point numbers, wherein the adder is comprised of a 1-bit signdata path, a 23-bit significand data path, and a 10-bit exponent datapath.
 7. The systolic circuit of claim 5, wherein the first inputcomprises an input data element and the second input comprises a reducedweight, wherein the first convertor is further configured to: receivethe first input and a weight; generate the first reduced input and thesecond input; and select the first reduced input or the second input tobe provided.
 8. The systolic circuit of claim 5, wherein the firstconvertor comprises: a trailing bit reducer configured to reduce aquantity of bits representing a significand portion of the first inputto produce a reduced significand portion of the first input; a rounderconfigured to round the reduced significand portion of the first inputbased on a remainder of the bits representing the significand portion ofthe first input not included within the reduced significand portion; andan exponent expander configured to increase a quantity of bitsrepresenting an exponent portion of the first input.
 9. The systoliccircuit of claim 5, wherein the first input comprises a first roundedinput, wherein the first convertor comprises: a trailing bit reducerconfigured to reduce a quantity of bits representing a significandportion of the first input to produce a reduced significand portion ofthe first input; and an exponent expander configured to increase aquantity of bits representing an exponent portion of the first input.10. The systolic circuit of claim 5, wherein the first reduced inputcomprises a first reduced rounded input, wherein the first reducedrounded input is rounded based on one or more of: stochastic rounding;rounding to nearest even; rounding to zero; rounding down; or roundingup.
 11. The systolic circuit of claim 5, wherein the first reduced inputcomprises a first reduced rounded input, wherein the first reducedrounded input is rounded based on a user input.
 12. The systolic circuitof claim 5, wherein: the first convertor is configured to convert 32-bitfloating-point numbers to 22-bit floating-point numbers, wherein each ofthe processing elements comprises: a 22-bit multiplier; and a 34-bitadder.
 13. The systolic circuit of claim 5, wherein: the first convertoris further configured to convert m-bit floating-point numbers to n-bitfloating-point numbers, wherein n and m can be any positive integer,wherein n is less than m, wherein each of the processing elementscomprises: a multiplier configured to multiply at least two n-bitnumbers; and an adder configured to add two p-bit numbers, wherein p isgreater than n.
 14. The systolic circuit of claim 5, wherein to reducethe quantity of trailing bits of the first input, the first convertor isconfigured to: set the quantity of trailing bits to zero.
 15. Thesystolic circuit of claim 5, further comprising: a second convertorconfigured to: receive a weight represented in floating-point with thefirst bit-length; identify a quantity of trailing bits of the weight;reduce the quantity of trailing bits of the weight; and generate thesecond input represented in floating-point with the second bit-lengthbased on reducing the quantity of trailing bits of the weight.
 16. Thesystolic circuit of claim 5, wherein the first reduced input is storedin a 24-bit format.
 17. A method implemented by a systolic circuit, themethod comprising: receiving a first input represented in floating-pointwith a first bit-length; reducing a quantity of trailing bits of thefirst input based on a difference between the first bit-length and abit-length supported by a processing element of the systolic circuit formultiply-accumulate operations; generating a first reduced inputrepresented in floating-point with a second bit-length based on reducingthe quantity of trailing bits of the first input, wherein the secondbit-length is less than the first bit-length, wherein the secondbit-length corresponds to the bit-length supported by the processingelement; and receiving the first reduced input and a second input forperforming, by the processing element, multiply-accumulate operations.18. The method of claim 17, wherein: the first input comprises a 32-bitfloating-point number; the first reduced input comprises a first 22-bitfloating-point number; and the second input comprises a second 22-bitfloating-point number.
 19. The method of claim 17, wherein generatingthe first reduced input comprises: rounding the first input to generatethe first reduced input, based on a remainder of non-trailing bits ofthe first input, wherein the first input comprises a quantity of bits,wherein rounding the first input comprises rounding a portion of thequantity of bits.
 20. The method of claim 17, wherein one or more of thefirst reduced input or the second input comprises a rounded, reducedinput, wherein the rounded, reduced input is rounded based on one ormore of: stochastic rounding; rounding to nearest even; rounding tozero; rounding down; or rounding up.
 21. The method of claim 17, whereinthe first bit-length is unsupported by the processing element for themultiply-accumulate operations.